[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435318278681-12427-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 15:08:40 dist-test-slave-865v krb5kdc[12434](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 15:08:40 dist-test-slave-865v krb5kdc[12434](info): set up 2 sockets
Sep 26 15:08:40 dist-test-slave-865v krb5kdc[12434](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260926 15:08:41.690905 12427 mini_kdc.cc:121] Time spent starting KDC: real 3.326s user 0.003s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435318278681-12427-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435318278681-12427-0/krb5kdc/test-user.keytab.
Sep 26 15:08:41 dist-test-slave-865v krb5kdc[12434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435321, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T15:08:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T15:08:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.34.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.34.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.34.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.254.keytab.
Entry for principal kudu/127.12.34.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.254.keytab.
I20260926 15:08:41.861207 12427 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8uoJGv/build/asan/bin/kudu
/tmp/dist-test-task8uoJGv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435318278681-12427-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435318278681-12427-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435318278681-12427-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.34.254:45575
--webserver_interface=127.12.34.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.254.keytab
--principal=kudu/127.12.34.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.34.212:46485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.34.254:45575
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435318278681-12427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435318278681-12427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435318278681-12427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:08:42.198347 12447 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:08:42.198632 12447 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:08:42.198797 12447 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:08:42.206651 12447 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:08:42.206765 12447 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:08:42.206877 12447 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:08:42.206957 12447 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:08:42.207036 12447 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:08:42.217001 12447 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.34.212:46485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435318278681-12427-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435318278681-12427-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.34.254:45575
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.254.keytab
--principal=kudu/127.12.34.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.34.254:45575
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.34.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435318278681-12427-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:08:42.218400 12447 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:08:42.219785 12447 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:08:42.232705 12453 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:08:42.232764 12455 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:08:42.232734 12452 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:08:42.233423 12447 server_base.cc:1034] running on GCE node
I20260926 15:08:42.234216 12447 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:08:42.235769 12447 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:08:42.239166 12447 hybrid_clock.cc:648] HybridClock initialized: now 1790435322239053 us; error 108 us; skew 500 ppm
I20260926 15:08:42.239563 12447 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:08:42 dist-test-slave-865v krb5kdc[12434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435322, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:08:42.243212 12447 init.cc:377] Logged in from keytab as kudu/127.12.34.254@KRBTEST.COM (short username kudu)
I20260926 15:08:42.244830 12447 webserver.cc:460] Webserver started at http://127.12.34.254:46631/ using document root <none> and password file <none>
I20260926 15:08:42.245635 12447 fs_manager.cc:359] Metadata directory not provided
I20260926 15:08:42.245776 12447 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:08:42.246083 12447 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:08:42.249166 12447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435318278681-12427-0/minicluster-data/master-0/data/instance:
uuid: "3bfebf7a4cab48ebb96e322d1810e34a"
format_stamp: "Formatted at 2026-09-26 15:08:42 on dist-test-slave-865v"
server_key: "7f1236701edc45f15ba2628da4e73170"
server_key_iv: "12afe8b70291b1e61b077c4754b821e4"
server_key_version: "encryptionkey@0"
I20260926 15:08:42.249934 12447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435318278681-12427-0/minicluster-data/master-0/wal/instance:
uuid: "3bfebf7a4cab48ebb96e322d1810e34a"
format_stamp: "Formatted at 2026-09-26 15:08:42 on dist-test-slave-865v"
server_key: "7f1236701edc45f15ba2628da4e73170"
server_key_iv: "12afe8b70291b1e61b077c4754b821e4"
server_key_version: "encryptionkey@0"
I20260926 15:08:42.255365 12447 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 15:08:42.258896 12462 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:08:42.260185 12447 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 15:08:42.260370 12447 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435318278681-12427-0/minicluster-data/master-0/data,/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435318278681-12427-0/minicluster-data/master-0/wal
uuid: "3bfebf7a4cab48ebb96e322d1810e34a"
format_stamp: "Formatted at 2026-09-26 15:08:42 on dist-test-slave-865v"
server_key: "7f1236701edc45f15ba2628da4e73170"
server_key_iv: "12afe8b70291b1e61b077c4754b821e4"
server_key_version: "encryptionkey@0"
I20260926 15:08:42.260567 12447 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435318278681-12427-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435318278681-12427-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435318278681-12427-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:08:42.273561 12447 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:08:42.277942 12447 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:08:42.278244 12447 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:08:42.298411 12447 rpc_server.cc:307] RPC server started. Bound to: 127.12.34.254:45575
I20260926 15:08:42.298453 12513 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.34.254:45575 every 8 connection(s)
I20260926 15:08:42.299785 12447 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
I20260926 15:08:42.303704 12514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:08:42.308447 12427 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8uoJGv/build/asan/bin/kudu as pid 12447
I20260926 15:08:42.308753 12427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435318278681-12427-0/minicluster-data/master-0/wal/instance
I20260926 15:08:42.310619 12427 external_mini_cluster.cc:1428] Setting key 55381c5a34f66fdb718848a78ecd1b5a
I20260926 15:08:42.313845 12514 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3bfebf7a4cab48ebb96e322d1810e34a: Bootstrap starting.
I20260926 15:08:42.317929 12514 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3bfebf7a4cab48ebb96e322d1810e34a: Neither blocks nor log segments found. Creating new log.
I20260926 15:08:42.318898 12514 log.cc:824] T 00000000000000000000000000000000 P 3bfebf7a4cab48ebb96e322d1810e34a: Log is configured to *not* fsync() on all Append() calls
I20260926 15:08:42.321553 12514 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3bfebf7a4cab48ebb96e322d1810e34a: No bootstrap required, opened a new log
I20260926 15:08:42.330427 12514 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3bfebf7a4cab48ebb96e322d1810e34a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bfebf7a4cab48ebb96e322d1810e34a" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 45575 } }
I20260926 15:08:42.330821 12514 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3bfebf7a4cab48ebb96e322d1810e34a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:08:42.330946 12514 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3bfebf7a4cab48ebb96e322d1810e34a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3bfebf7a4cab48ebb96e322d1810e34a, State: Initialized, Role: FOLLOWER
I20260926 15:08:42.331565 12514 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3bfebf7a4cab48ebb96e322d1810e34a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bfebf7a4cab48ebb96e322d1810e34a" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 45575 } }
I20260926 15:08:42.331789 12514 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3bfebf7a4cab48ebb96e322d1810e34a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 15:08:42.331928 12514 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3bfebf7a4cab48ebb96e322d1810e34a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 15:08:42.332113 12514 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3bfebf7a4cab48ebb96e322d1810e34a [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:08:42.334327 12514 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3bfebf7a4cab48ebb96e322d1810e34a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bfebf7a4cab48ebb96e322d1810e34a" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 45575 } }
I20260926 15:08:42.334828 12514 leader_election.cc:302] T 00000000000000000000000000000000 P 3bfebf7a4cab48ebb96e322d1810e34a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3bfebf7a4cab48ebb96e322d1810e34a; no voters:
I20260926 15:08:42.335923 12514 leader_election.cc:288] T 00000000000000000000000000000000 P 3bfebf7a4cab48ebb96e322d1810e34a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 15:08:42.336030 12519 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3bfebf7a4cab48ebb96e322d1810e34a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:08:42.337216 12519 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3bfebf7a4cab48ebb96e322d1810e34a [term 1 LEADER]: Becoming Leader. State: Replica: 3bfebf7a4cab48ebb96e322d1810e34a, State: Running, Role: LEADER
I20260926 15:08:42.337796 12519 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3bfebf7a4cab48ebb96e322d1810e34a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bfebf7a4cab48ebb96e322d1810e34a" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 45575 } }
I20260926 15:08:42.338367 12514 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3bfebf7a4cab48ebb96e322d1810e34a [sys.catalog]: configured and running, proceeding with master startup.
Sep 26 15:08:42 dist-test-slave-865v krb5kdc[12434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435321, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.34.254@KRBTEST.COM
I20260926 15:08:42.340695 12520 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3bfebf7a4cab48ebb96e322d1810e34a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3bfebf7a4cab48ebb96e322d1810e34a. Latest consensus state: current_term: 1 leader_uuid: "3bfebf7a4cab48ebb96e322d1810e34a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bfebf7a4cab48ebb96e322d1810e34a" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 45575 } } }
I20260926 15:08:42.341138 12520 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3bfebf7a4cab48ebb96e322d1810e34a [sys.catalog]: This master's current role is: LEADER
I20260926 15:08:42.345508 12521 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3bfebf7a4cab48ebb96e322d1810e34a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3bfebf7a4cab48ebb96e322d1810e34a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bfebf7a4cab48ebb96e322d1810e34a" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 45575 } } }
I20260926 15:08:42.347191 12521 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3bfebf7a4cab48ebb96e322d1810e34a [sys.catalog]: This master's current role is: LEADER
I20260926 15:08:42.349673 12524 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:08:42.353526 12516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:08:42.314377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35506 (local address 127.12.34.254:45575)
0926 15:08:42.315221 (+ 844us) server_negotiation.cc:206] Beginning negotiation
0926 15:08:42.315241 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 15:08:42.316810 (+ 1569us) server_negotiation.cc:407] Connection header received
0926 15:08:42.326566 (+ 9756us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:08:42.326588 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:08:42.327015 (+ 427us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:08:42.327468 (+ 453us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:08:42.330396 (+ 2928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:08:42.332083 (+ 1687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:08:42.333806 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:08:42.334501 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:08:42.334670 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:08:42.341450 (+ 6780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:08:42.341462 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:08:42.341480 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:08:42.341525 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:08:42.347093 (+ 5568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:08:42.348360 (+ 1267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:08:42.348368 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:08:42.348376 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:08:42.348577 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:08:42.349054 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:08:42.349058 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:08:42.349060 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:08:42.349874 (+ 814us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:08:42.351010 (+ 1136us) server_negotiation.cc:1035] Waiting for connection context
0926 15:08:42.351313 (+ 303us) server_negotiation.cc:299] Negotiation successful
0926 15:08:42.351817 (+ 504us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":604,"thread_start_us":409,"threads_started":1}
I20260926 15:08:42.358590 12524 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 15:08:42.367996 12524 catalog_manager.cc:1347] Generated new cluster ID: cc2799fdd36a4506855034e77fc25573
I20260926 15:08:42.368114 12524 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 15:08:42.384934 12524 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 15:08:42.386233 12524 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:08:42.395216 12524 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3bfebf7a4cab48ebb96e322d1810e34a: Generated new TSK 0
I20260926 15:08:42.396247 12524 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.34.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.34.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.34.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.193.keytab.
Entry for principal kudu/127.12.34.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.193.keytab.
I20260926 15:08:42.444284 12427 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8uoJGv/build/asan/bin/kudu
/tmp/dist-test-task8uoJGv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435318278681-12427-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435318278681-12427-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435318278681-12427-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435318278681-12427-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.34.193:0
--local_ip_for_outbound_sockets=127.12.34.193
--webserver_interface=127.12.34.193
--webserver_port=0
--tserver_master_addrs=127.12.34.254:45575
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.193.keytab
--principal=kudu/127.12.34.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.34.212:46485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435318278681-12427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435318278681-12427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435318278681-12427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:08:42.779059 12540 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:08:42.779321 12540 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:08:42.779477 12540 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:08:42.787472 12540 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:08:42.787580 12540 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:08:42.787751 12540 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.34.193
I20260926 15:08:42.797068 12540 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.34.212:46485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435318278681-12427-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435318278681-12427-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.193.keytab
--principal=kudu/127.12.34.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.34.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435318278681-12427-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.34.193
--webserver_port=0
--tserver_master_addrs=127.12.34.254:45575
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.34.193
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435318278681-12427-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:08:42.798317 12540 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:08:42.799618 12540 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:08:42.812479 12546 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:08:42.812489 12545 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:08:42.812520 12548 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:08:42.813112 12540 server_base.cc:1034] running on GCE node
I20260926 15:08:42.813669 12540 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:08:42.814525 12540 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:08:42.815745 12540 hybrid_clock.cc:648] HybridClock initialized: now 1790435322815713 us; error 51 us; skew 500 ppm
I20260926 15:08:42.816045 12540 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:08:42 dist-test-slave-865v krb5kdc[12434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435322, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:08:42.818943 12540 init.cc:377] Logged in from keytab as kudu/127.12.34.193@KRBTEST.COM (short username kudu)
I20260926 15:08:42.820389 12540 webserver.cc:460] Webserver started at http://127.12.34.193:46043/ using document root <none> and password file <none>
I20260926 15:08:42.821013 12540 fs_manager.cc:359] Metadata directory not provided
I20260926 15:08:42.821190 12540 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:08:42.821492 12540 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:08:42.824532 12540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435318278681-12427-0/minicluster-data/ts-0/data/instance:
uuid: "cc55c2d5f9df4f7ea863c483ecefa5f5"
format_stamp: "Formatted at 2026-09-26 15:08:42 on dist-test-slave-865v"
server_key: "7b4777b7e1a437f485c5910e13227a57"
server_key_iv: "4be629ad5db7928ee8fde9e50fc5f873"
server_key_version: "encryptionkey@0"
I20260926 15:08:42.825389 12540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435318278681-12427-0/minicluster-data/ts-0/wal/instance:
uuid: "cc55c2d5f9df4f7ea863c483ecefa5f5"
format_stamp: "Formatted at 2026-09-26 15:08:42 on dist-test-slave-865v"
server_key: "7b4777b7e1a437f485c5910e13227a57"
server_key_iv: "4be629ad5db7928ee8fde9e50fc5f873"
server_key_version: "encryptionkey@0"
I20260926 15:08:42.830204 12540 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.002s
I20260926 15:08:42.833415 12555 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:08:42.834578 12540 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 15:08:42.834740 12540 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435318278681-12427-0/minicluster-data/ts-0/data,/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435318278681-12427-0/minicluster-data/ts-0/wal
uuid: "cc55c2d5f9df4f7ea863c483ecefa5f5"
format_stamp: "Formatted at 2026-09-26 15:08:42 on dist-test-slave-865v"
server_key: "7b4777b7e1a437f485c5910e13227a57"
server_key_iv: "4be629ad5db7928ee8fde9e50fc5f873"
server_key_version: "encryptionkey@0"
I20260926 15:08:42.834931 12540 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435318278681-12427-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435318278681-12427-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435318278681-12427-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:08:42.845660 12540 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:08:42.849309 12540 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:08:42.849575 12540 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:08:42.850497 12540 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:08:42.851756 12540 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:08:42.851854 12540 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:08:42.851971 12540 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:08:42.852049 12540 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:08:42.887634 12540 rpc_server.cc:307] RPC server started. Bound to: 127.12.34.193:45539
I20260926 15:08:42.887678 12667 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.34.193:45539 every 8 connection(s)
I20260926 15:08:42.888890 12540 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435318278681-12427-0/minicluster-data/ts-0/data/info.pb
I20260926 15:08:42.891211 12427 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8uoJGv/build/asan/bin/kudu as pid 12540
I20260926 15:08:42.891453 12427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435318278681-12427-0/minicluster-data/ts-0/wal/instance
I20260926 15:08:42.892148 12427 external_mini_cluster.cc:1428] Setting key 516d5d9dcb8e1ddeafefbb243908507d
Sep 26 15:08:42 dist-test-slave-865v krb5kdc[12434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435322, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.193@KRBTEST.COM for kudu/127.12.34.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.12.34.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.34.194@KRBTEST.COM" created.
I20260926 15:08:42.912029 12670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:08:42.891535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.193:56581 (local address 127.12.34.254:45575)
0926 15:08:42.891974 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0926 15:08:42.891980 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:08:42.895512 (+ 3532us) server_negotiation.cc:407] Connection header received
0926 15:08:42.895578 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:08:42.895586 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:08:42.895675 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:08:42.895788 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:08:42.897346 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:08:42.898490 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:08:42.900478 (+ 1988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:08:42.901287 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:08:42.901434 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:08:42.905527 (+ 4093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:08:42.905535 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:08:42.905542 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:08:42.905589 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:08:42.909138 (+ 3549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:08:42.909813 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:08:42.909817 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:08:42.909819 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:08:42.909926 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:08:42.910341 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:08:42.910345 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:08:42.910346 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:08:42.910596 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:08:42.910702 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 15:08:42.911470 (+ 768us) server_negotiation.cc:299] Negotiation successful
0926 15:08:42.911691 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":202,"threads_started":1}
I20260926 15:08:42.913053 12671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:08:42.892116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:45575 (local address 127.12.34.193:56581)
0926 15:08:42.892907 (+ 791us) negotiation.cc:107] Waiting for socket to connect
0926 15:08:42.892950 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 15:08:42.895129 (+ 2179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:08:42.896038 (+ 909us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:08:42.896046 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:08:42.896545 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:08:42.897185 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:08:42.897204 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:08:42.898649 (+ 1445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:08:42.898658 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:08:42.900315 (+ 1657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:08:42.900331 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:08:42.901501 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:08:42.901506 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:08:42.901697 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:08:42.902403 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:08:42.902431 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:08:42.905298 (+ 2867us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:08:42.909264 (+ 3966us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:08:42.909271 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:08:42.909284 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:08:42.909699 (+ 415us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:08:42.910043 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:08:42.910047 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:08:42.910049 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:08:42.910227 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:08:42.910711 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:08:42.910718 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:08:42.911139 (+ 421us) client_negotiation.cc:769] Sending connection context
0926 15:08:42.911361 (+ 222us) client_negotiation.cc:240] Negotiation successful
0926 15:08:42.911920 (+ 559us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":297,"threads_started":1}
I20260926 15:08:42.914740 12668 heartbeater.cc:347] Connected to a master server at 127.12.34.254:45575
I20260926 15:08:42.915084 12668 heartbeater.cc:464] Registering TS with master...
I20260926 15:08:42.915989 12668 heartbeater.cc:511] Master 127.12.34.254:45575 requested a full tablet report, sending...
I20260926 15:08:42.918224 12479 ts_manager.cc:194] Registered new tserver with Master: cc55c2d5f9df4f7ea863c483ecefa5f5 (127.12.34.193:45539)
I20260926 15:08:42.919704 12479 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.34.193@KRBTEST.COM'} at 127.12.34.193:56581
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.34.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.194.keytab.
Entry for principal kudu/127.12.34.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.194.keytab.
I20260926 15:08:42.932320 12427 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8uoJGv/build/asan/bin/kudu
/tmp/dist-test-task8uoJGv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435318278681-12427-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435318278681-12427-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435318278681-12427-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435318278681-12427-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.34.194:0
--local_ip_for_outbound_sockets=127.12.34.194
--webserver_interface=127.12.34.194
--webserver_port=0
--tserver_master_addrs=127.12.34.254:45575
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.194.keytab
--principal=kudu/127.12.34.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.34.212:46485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435318278681-12427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435318278681-12427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435318278681-12427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:08:43.255393 12674 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:08:43.255672 12674 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:08:43.255775 12674 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:08:43.263230 12674 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:08:43.263361 12674 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:08:43.263932 12674 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.34.194
I20260926 15:08:43.273547 12674 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.34.212:46485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435318278681-12427-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435318278681-12427-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.194.keytab
--principal=kudu/127.12.34.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.34.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435318278681-12427-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.34.194
--webserver_port=0
--tserver_master_addrs=127.12.34.254:45575
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.34.194
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435318278681-12427-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:08:43.274950 12674 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:08:43.276327 12674 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:08:43.293524 12680 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:08:43.293576 12679 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:08:43.293571 12682 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:08:43.294227 12674 server_base.cc:1034] running on GCE node
I20260926 15:08:43.294873 12674 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:08:43.295734 12674 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:08:43.296985 12674 hybrid_clock.cc:648] HybridClock initialized: now 1790435323296931 us; error 55 us; skew 500 ppm
I20260926 15:08:43.297329 12674 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:08:43 dist-test-slave-865v krb5kdc[12434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435323, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:08:43.300381 12674 init.cc:377] Logged in from keytab as kudu/127.12.34.194@KRBTEST.COM (short username kudu)
I20260926 15:08:43.301956 12674 webserver.cc:460] Webserver started at http://127.12.34.194:44657/ using document root <none> and password file <none>
I20260926 15:08:43.302599 12674 fs_manager.cc:359] Metadata directory not provided
I20260926 15:08:43.302723 12674 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:08:43.303048 12674 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:08:43.306244 12674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435318278681-12427-0/minicluster-data/ts-1/data/instance:
uuid: "338f935c05ff4fc98d6bcefc2d422f12"
format_stamp: "Formatted at 2026-09-26 15:08:43 on dist-test-slave-865v"
server_key: "7e728bce7a89ca7c62a36b20173bb59a"
server_key_iv: "2119ae063241d2cab95558b53ac4665d"
server_key_version: "encryptionkey@0"
I20260926 15:08:43.306989 12674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435318278681-12427-0/minicluster-data/ts-1/wal/instance:
uuid: "338f935c05ff4fc98d6bcefc2d422f12"
format_stamp: "Formatted at 2026-09-26 15:08:43 on dist-test-slave-865v"
server_key: "7e728bce7a89ca7c62a36b20173bb59a"
server_key_iv: "2119ae063241d2cab95558b53ac4665d"
server_key_version: "encryptionkey@0"
I20260926 15:08:43.312206 12674 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260926 15:08:43.315759 12689 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:08:43.317647 12674 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 15:08:43.317806 12674 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435318278681-12427-0/minicluster-data/ts-1/data,/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435318278681-12427-0/minicluster-data/ts-1/wal
uuid: "338f935c05ff4fc98d6bcefc2d422f12"
format_stamp: "Formatted at 2026-09-26 15:08:43 on dist-test-slave-865v"
server_key: "7e728bce7a89ca7c62a36b20173bb59a"
server_key_iv: "2119ae063241d2cab95558b53ac4665d"
server_key_version: "encryptionkey@0"
I20260926 15:08:43.318027 12674 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435318278681-12427-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435318278681-12427-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435318278681-12427-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:08:43.331372 12674 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:08:43.337009 12674 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:08:43.337370 12674 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:08:43.338373 12674 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:08:43.339834 12674 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:08:43.339943 12674 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:08:43.340085 12674 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:08:43.340143 12674 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:08:43.377507 12674 rpc_server.cc:307] RPC server started. Bound to: 127.12.34.194:40615
I20260926 15:08:43.377537 12801 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.34.194:40615 every 8 connection(s)
I20260926 15:08:43.378741 12674 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435318278681-12427-0/minicluster-data/ts-1/data/info.pb
I20260926 15:08:43.380229 12427 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8uoJGv/build/asan/bin/kudu as pid 12674
I20260926 15:08:43.380421 12427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435318278681-12427-0/minicluster-data/ts-1/wal/instance
I20260926 15:08:43.380841 12427 external_mini_cluster.cc:1428] Setting key 5458a1e450a3e0564889410a3d119fb0
Sep 26 15:08:43 dist-test-slave-865v krb5kdc[12434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435323, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.194@KRBTEST.COM for kudu/127.12.34.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.12.34.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.34.195@KRBTEST.COM" created.
I20260926 15:08:43.402300 12670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:08:43.382526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.194:50333 (local address 127.12.34.254:45575)
0926 15:08:43.382737 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0926 15:08:43.382744 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:08:43.384497 (+ 1753us) server_negotiation.cc:407] Connection header received
0926 15:08:43.386570 (+ 2073us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:08:43.386577 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:08:43.386668 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:08:43.386780 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:08:43.388376 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:08:43.389208 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:08:43.391764 (+ 2556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:08:43.392172 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:08:43.392271 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:08:43.396542 (+ 4271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:08:43.396547 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:08:43.396550 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:08:43.396580 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:08:43.399254 (+ 2674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:08:43.399978 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:08:43.399985 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:08:43.399988 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:08:43.400112 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:08:43.400595 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:08:43.400599 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:08:43.400601 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:08:43.400855 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:08:43.400980 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0926 15:08:43.401896 (+ 916us) server_negotiation.cc:299] Negotiation successful
0926 15:08:43.402069 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 15:08:43.403896 12804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:08:43.383111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:45575 (local address 127.12.34.194:50333)
0926 15:08:43.384324 (+ 1213us) negotiation.cc:107] Waiting for socket to connect
0926 15:08:43.384374 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 15:08:43.386106 (+ 1732us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:08:43.387092 (+ 986us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:08:43.387102 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:08:43.387650 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:08:43.388191 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:08:43.388207 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:08:43.389365 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:08:43.389373 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:08:43.391586 (+ 2213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:08:43.391603 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:08:43.392397 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:08:43.392406 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:08:43.392718 (+ 312us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:08:43.393581 (+ 863us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:08:43.393615 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:08:43.396301 (+ 2686us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:08:43.399419 (+ 3118us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:08:43.399427 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:08:43.399444 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:08:43.399855 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:08:43.400222 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:08:43.400228 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:08:43.400232 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:08:43.400456 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:08:43.400998 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:08:43.401009 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:08:43.401503 (+ 494us) client_negotiation.cc:769] Sending connection context
0926 15:08:43.401824 (+ 321us) client_negotiation.cc:240] Negotiation successful
0926 15:08:43.402412 (+ 588us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":964,"thread_start_us":319,"threads_started":1}
I20260926 15:08:43.405560 12802 heartbeater.cc:347] Connected to a master server at 127.12.34.254:45575
I20260926 15:08:43.405869 12802 heartbeater.cc:464] Registering TS with master...
I20260926 15:08:43.406600 12802 heartbeater.cc:511] Master 127.12.34.254:45575 requested a full tablet report, sending...
I20260926 15:08:43.408264 12479 ts_manager.cc:194] Registered new tserver with Master: 338f935c05ff4fc98d6bcefc2d422f12 (127.12.34.194:40615)
I20260926 15:08:43.409047 12479 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.34.194@KRBTEST.COM'} at 127.12.34.194:50333
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.34.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.195.keytab.
Entry for principal kudu/127.12.34.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.195.keytab.
I20260926 15:08:43.422032 12427 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8uoJGv/build/asan/bin/kudu
/tmp/dist-test-task8uoJGv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435318278681-12427-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435318278681-12427-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435318278681-12427-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435318278681-12427-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.34.195:0
--local_ip_for_outbound_sockets=127.12.34.195
--webserver_interface=127.12.34.195
--webserver_port=0
--tserver_master_addrs=127.12.34.254:45575
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.195.keytab
--principal=kudu/127.12.34.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.34.212:46485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435318278681-12427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435318278681-12427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435318278681-12427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:08:43.740056 12807 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:08:43.740334 12807 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:08:43.740499 12807 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:08:43.747665 12807 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:08:43.747767 12807 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:08:43.747959 12807 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.34.195
I20260926 15:08:43.757576 12807 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.34.212:46485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435318278681-12427-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435318278681-12427-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.195.keytab
--principal=kudu/127.12.34.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.34.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435318278681-12427-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.34.195
--webserver_port=0
--tserver_master_addrs=127.12.34.254:45575
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.34.195
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435318278681-12427-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:08:43.758817 12807 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:08:43.760136 12807 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:08:43.773237 12812 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:08:43.773285 12815 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:08:43.773458 12813 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:08:43.774509 12807 server_base.cc:1034] running on GCE node
I20260926 15:08:43.775089 12807 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:08:43.775990 12807 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:08:43.777197 12807 hybrid_clock.cc:648] HybridClock initialized: now 1790435323777151 us; error 32 us; skew 500 ppm
I20260926 15:08:43.777544 12807 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:08:43 dist-test-slave-865v krb5kdc[12434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435323, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:08:43.781039 12807 init.cc:377] Logged in from keytab as kudu/127.12.34.195@KRBTEST.COM (short username kudu)
I20260926 15:08:43.782466 12807 webserver.cc:460] Webserver started at http://127.12.34.195:33283/ using document root <none> and password file <none>
I20260926 15:08:43.783113 12807 fs_manager.cc:359] Metadata directory not provided
I20260926 15:08:43.783233 12807 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:08:43.783545 12807 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:08:43.786422 12807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435318278681-12427-0/minicluster-data/ts-2/data/instance:
uuid: "255f90a2d3144ecfb287875be8dabdab"
format_stamp: "Formatted at 2026-09-26 15:08:43 on dist-test-slave-865v"
server_key: "8497cc2bcf830dad25b59498992f6af6"
server_key_iv: "6d9a6d7e0b2ad2f0909907afd76b81d2"
server_key_version: "encryptionkey@0"
I20260926 15:08:43.787146 12807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435318278681-12427-0/minicluster-data/ts-2/wal/instance:
uuid: "255f90a2d3144ecfb287875be8dabdab"
format_stamp: "Formatted at 2026-09-26 15:08:43 on dist-test-slave-865v"
server_key: "8497cc2bcf830dad25b59498992f6af6"
server_key_iv: "6d9a6d7e0b2ad2f0909907afd76b81d2"
server_key_version: "encryptionkey@0"
I20260926 15:08:43.791977 12807 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 15:08:43.795298 12822 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:08:43.796568 12807 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260926 15:08:43.796734 12807 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435318278681-12427-0/minicluster-data/ts-2/data,/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435318278681-12427-0/minicluster-data/ts-2/wal
uuid: "255f90a2d3144ecfb287875be8dabdab"
format_stamp: "Formatted at 2026-09-26 15:08:43 on dist-test-slave-865v"
server_key: "8497cc2bcf830dad25b59498992f6af6"
server_key_iv: "6d9a6d7e0b2ad2f0909907afd76b81d2"
server_key_version: "encryptionkey@0"
I20260926 15:08:43.796895 12807 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435318278681-12427-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435318278681-12427-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435318278681-12427-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:08:43.814582 12807 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:08:43.818776 12807 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:08:43.819053 12807 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:08:43.819952 12807 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:08:43.821316 12807 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:08:43.821386 12807 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:08:43.821539 12807 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:08:43.821602 12807 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:08:43.856814 12807 rpc_server.cc:307] RPC server started. Bound to: 127.12.34.195:46651
I20260926 15:08:43.856858 12934 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.34.195:46651 every 8 connection(s)
I20260926 15:08:43.858139 12807 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435318278681-12427-0/minicluster-data/ts-2/data/info.pb
I20260926 15:08:43.867453 12427 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8uoJGv/build/asan/bin/kudu as pid 12807
I20260926 15:08:43.867648 12427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790435318278681-12427-0/minicluster-data/ts-2/wal/instance
I20260926 15:08:43.868189 12427 external_mini_cluster.cc:1428] Setting key aebde601e5a927870f9fbeb2b30540dc
Sep 26 15:08:43 dist-test-slave-865v krb5kdc[12434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435323, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.195@KRBTEST.COM for kudu/127.12.34.254@KRBTEST.COM
I20260926 15:08:43.878700 12670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:08:43.860570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.195:32783 (local address 127.12.34.254:45575)
0926 15:08:43.861189 (+ 619us) server_negotiation.cc:206] Beginning negotiation
0926 15:08:43.861196 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:08:43.862424 (+ 1228us) server_negotiation.cc:407] Connection header received
0926 15:08:43.864288 (+ 1864us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:08:43.864292 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:08:43.864364 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:08:43.864465 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:08:43.865998 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:08:43.866846 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:08:43.869129 (+ 2283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:08:43.869650 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:08:43.869785 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:08:43.873564 (+ 3779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:08:43.873569 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:08:43.873572 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:08:43.873599 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:08:43.875881 (+ 2282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:08:43.876577 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:08:43.876584 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:08:43.876587 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:08:43.876708 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:08:43.877125 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:08:43.877130 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:08:43.877133 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:08:43.877375 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:08:43.877522 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0926 15:08:43.878163 (+ 641us) server_negotiation.cc:299] Negotiation successful
0926 15:08:43.878362 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":450}
I20260926 15:08:43.879573 12937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:08:43.861609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:45575 (local address 127.12.34.195:32783)
0926 15:08:43.862260 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0926 15:08:43.862302 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 15:08:43.863989 (+ 1687us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:08:43.864664 (+ 675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:08:43.864671 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:08:43.865300 (+ 629us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:08:43.865831 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:08:43.865847 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:08:43.866975 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:08:43.866978 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:08:43.868957 (+ 1979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:08:43.868971 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:08:43.869912 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:08:43.869922 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:08:43.870151 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:08:43.870917 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:08:43.870944 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:08:43.873362 (+ 2418us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:08:43.876049 (+ 2687us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:08:43.876057 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:08:43.876072 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:08:43.876449 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:08:43.876811 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:08:43.876817 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:08:43.876821 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:08:43.876994 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:08:43.877504 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:08:43.877511 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:08:43.877802 (+ 291us) client_negotiation.cc:769] Sending connection context
0926 15:08:43.878049 (+ 247us) client_negotiation.cc:240] Negotiation successful
0926 15:08:43.878542 (+ 493us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":252,"threads_started":1}
I20260926 15:08:43.880842 12935 heartbeater.cc:347] Connected to a master server at 127.12.34.254:45575
I20260926 15:08:43.881155 12935 heartbeater.cc:464] Registering TS with master...
I20260926 15:08:43.882263 12935 heartbeater.cc:511] Master 127.12.34.254:45575 requested a full tablet report, sending...
I20260926 15:08:43.884028 12479 ts_manager.cc:194] Registered new tserver with Master: 255f90a2d3144ecfb287875be8dabdab (127.12.34.195:46651)
I20260926 15:08:43.884912 12479 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.34.195@KRBTEST.COM'} at 127.12.34.195:32783
I20260926 15:08:43.896292 12427 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 15:08:43 dist-test-slave-865v krb5kdc[12434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435323, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
I20260926 15:08:43.922680 12668 heartbeater.cc:503] Master 127.12.34.254:45575 was elected leader, sending a full tablet report...
Sep 26 15:08:43 dist-test-slave-865v krb5kdc[12434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435323, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.12.34.193@KRBTEST.COM
I20260926 15:08:43.942945 12942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:08:43.925143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55632 (local address 127.12.34.193:45539)
0926 15:08:43.926643 (+ 1500us) server_negotiation.cc:206] Beginning negotiation
0926 15:08:43.926650 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:08:43.926673 (+ 23us) server_negotiation.cc:407] Connection header received
0926 15:08:43.926722 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:08:43.926728 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:08:43.926935 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:08:43.927049 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:08:43.928573 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:08:43.929665 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:08:43.931000 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:08:43.931532 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:08:43.931625 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:08:43.935978 (+ 4353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:08:43.935993 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:08:43.936008 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:08:43.936052 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:08:43.939033 (+ 2981us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:08:43.939860 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:08:43.939873 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:08:43.939876 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:08:43.940032 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:08:43.940511 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:08:43.940517 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:08:43.940520 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:08:43.941055 (+ 535us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:08:43.941408 (+ 353us) server_negotiation.cc:1035] Waiting for connection context
0926 15:08:43.942275 (+ 867us) server_negotiation.cc:299] Negotiation successful
0926 15:08:43.942549 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1359,"thread_start_us":1218,"threads_started":1}
W20260926 15:08:43.944161 12583 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55632
Sep 26 15:08:43 dist-test-slave-865v krb5kdc[12434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435323, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 15:08:43 dist-test-slave-865v krb5kdc[12434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435323, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.34.193@KRBTEST.COM
I20260926 15:08:43.987756 12942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:08:43.974035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55646 (local address 127.12.34.193:45539)
0926 15:08:43.974222 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0926 15:08:43.974227 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:08:43.974584 (+ 357us) server_negotiation.cc:407] Connection header received
0926 15:08:43.974715 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:08:43.974719 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:08:43.974783 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:08:43.974873 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:08:43.976129 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:08:43.976829 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:08:43.978125 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:08:43.978570 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:08:43.978678 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:08:43.982866 (+ 4188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:08:43.982870 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:08:43.982873 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:08:43.982903 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:08:43.985561 (+ 2658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:08:43.986220 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:08:43.986223 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:08:43.986226 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:08:43.986336 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:08:43.986715 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:08:43.986718 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:08:43.986720 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:08:43.986940 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:08:43.987111 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0926 15:08:43.987383 (+ 272us) server_negotiation.cc:299] Negotiation successful
0926 15:08:43.987531 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 15:08:43.989194 12427 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 12540
I20260926 15:08:44.001929 12942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:08:43.990063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55650 (local address 127.12.34.193:45539)
0926 15:08:43.990261 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0926 15:08:43.990269 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 15:08:43.990532 (+ 263us) server_negotiation.cc:407] Connection header received
0926 15:08:43.990647 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:08:43.990650 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:08:43.990700 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:08:43.990774 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:08:43.992081 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:08:43.992778 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:08:43.994210 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:08:43.994653 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:08:43.994749 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:08:43.996889 (+ 2140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:08:43.996894 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:08:43.996897 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:08:43.996925 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:08:43.999346 (+ 2421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:08:43.999963 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:08:43.999997 (+ 34us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:08:44.000006 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:08:44.000151 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:08:44.000555 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:08:44.000563 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:08:44.000566 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:08:44.000814 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:08:44.001067 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0926 15:08:44.001385 (+ 318us) server_negotiation.cc:299] Negotiation successful
0926 15:08:44.001581 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 15:08:44.002609 12663 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:08:44.086949 12427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8uoJGv/build/asan/bin/kudu with pid 12540
I20260926 15:08:44.102874 12427 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 12674
Sep 26 15:08:44 dist-test-slave-865v krb5kdc[12434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435323, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.34.194@KRBTEST.COM
I20260926 15:08:44.118827 12949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:08:44.103775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36938 (local address 127.12.34.194:40615)
0926 15:08:44.104326 (+ 551us) server_negotiation.cc:206] Beginning negotiation
0926 15:08:44.104333 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:08:44.104359 (+ 26us) server_negotiation.cc:407] Connection header received
0926 15:08:44.104419 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:08:44.104426 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:08:44.104640 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:08:44.104766 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:08:44.105975 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:08:44.106996 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:08:44.108252 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:08:44.108777 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:08:44.108890 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:08:44.113036 (+ 4146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:08:44.113049 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:08:44.113072 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:08:44.113169 (+ 97us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:08:44.116285 (+ 3116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:08:44.116972 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:08:44.116981 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:08:44.116984 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:08:44.117190 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:08:44.117562 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:08:44.117565 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:08:44.117567 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:08:44.117876 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:08:44.118110 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0926 15:08:44.118334 (+ 224us) server_negotiation.cc:299] Negotiation successful
0926 15:08:44.118527 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":297,"threads_started":1}
I20260926 15:08:44.120031 12797 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:08:44.200624 12427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8uoJGv/build/asan/bin/kudu with pid 12674
I20260926 15:08:44.222518 12427 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 12807
Sep 26 15:08:44 dist-test-slave-865v krb5kdc[12434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435323, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.34.195@KRBTEST.COM
I20260926 15:08:44.239169 12951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:08:44.223621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36444 (local address 127.12.34.195:46651)
0926 15:08:44.224240 (+ 619us) server_negotiation.cc:206] Beginning negotiation
0926 15:08:44.224247 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:08:44.224276 (+ 29us) server_negotiation.cc:407] Connection header received
0926 15:08:44.224340 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:08:44.224346 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:08:44.224554 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:08:44.224751 (+ 197us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:08:44.226092 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:08:44.227087 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:08:44.228371 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:08:44.228920 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:08:44.229062 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:08:44.233167 (+ 4105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:08:44.233178 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:08:44.233193 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:08:44.233225 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:08:44.236139 (+ 2914us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:08:44.236781 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:08:44.236788 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:08:44.236790 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:08:44.236919 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:08:44.237301 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:08:44.237308 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:08:44.237314 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:08:44.237685 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:08:44.237915 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0926 15:08:44.238625 (+ 710us) server_negotiation.cc:299] Negotiation successful
0926 15:08:44.238835 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":482,"thread_start_us":317,"threads_started":1}
I20260926 15:08:44.240530 12930 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:08:44.324867 12427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8uoJGv/build/asan/bin/kudu with pid 12807
I20260926 15:08:44.340781 12427 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 12447
I20260926 15:08:44.342021 12509 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:08:44.441570 12427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8uoJGv/build/asan/bin/kudu with pid 12447
2026-09-26T15:08:44Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (6128 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435318278681-12427-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 15:08:44 dist-test-slave-865v krb5kdc[12956](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 15:08:44 dist-test-slave-865v krb5kdc[12956](info): set up 2 sockets
Sep 26 15:08:44 dist-test-slave-865v krb5kdc[12956](info): commencing operation
krb5kdc: starting...
W20260926 15:08:45.521018 12427 mini_kdc.cc:121] Time spent starting KDC: real 1.038s user 0.003s sys 0.010s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435318278681-12427-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435318278681-12427-0/krb5kdc/test-user.keytab.
Sep 26 15:08:45 dist-test-slave-865v krb5kdc[12956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435325, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T15:08:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T15:08:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.34.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.34.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.34.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.254.keytab.
Entry for principal kudu/127.12.34.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.254.keytab.
I20260926 15:08:45.693005 12427 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8uoJGv/build/asan/bin/kudu
/tmp/dist-test-task8uoJGv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435318278681-12427-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435318278681-12427-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435318278681-12427-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.34.254:35853
--webserver_interface=127.12.34.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.254.keytab
--principal=kudu/127.12.34.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.34.212:46693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.34.254:35853
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435318278681-12427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435318278681-12427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435318278681-12427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:08:45.999557 12969 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:08:45.999820 12969 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:08:45.999975 12969 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:08:46.007004 12969 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:08:46.007102 12969 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:08:46.007189 12969 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:08:46.007315 12969 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:08:46.007426 12969 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:08:46.016808 12969 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.34.212:46693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435318278681-12427-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435318278681-12427-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.34.254:35853
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.254.keytab
--principal=kudu/127.12.34.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.34.254:35853
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.34.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435318278681-12427-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:08:46.018115 12969 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:08:46.019378 12969 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:08:46.030993 12974 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:08:46.031114 12975 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:08:46.031446 12977 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:08:46.031800 12969 server_base.cc:1034] running on GCE node
I20260926 15:08:46.032548 12969 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:08:46.033950 12969 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:08:46.035144 12969 hybrid_clock.cc:648] HybridClock initialized: now 1790435326035090 us; error 54 us; skew 500 ppm
I20260926 15:08:46.035491 12969 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:08:46 dist-test-slave-865v krb5kdc[12956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435326, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:08:46.042605 12969 init.cc:377] Logged in from keytab as kudu/127.12.34.254@KRBTEST.COM (short username kudu)
I20260926 15:08:46.044005 12969 webserver.cc:460] Webserver started at http://127.12.34.254:45813/ using document root <none> and password file <none>
I20260926 15:08:46.044620 12969 fs_manager.cc:359] Metadata directory not provided
I20260926 15:08:46.044714 12969 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:08:46.044955 12969 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:08:46.048049 12969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435318278681-12427-0/minicluster-data/master-0/data/instance:
uuid: "79ce11dccdfe41ac9175b32411225ac3"
format_stamp: "Formatted at 2026-09-26 15:08:46 on dist-test-slave-865v"
server_key: "7900e656bbe5e7e49e75ee966ca64a10"
server_key_iv: "2ad8876a74fc471b9eae8882c9c3ab12"
server_key_version: "encryptionkey@0"
I20260926 15:08:46.048722 12969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435318278681-12427-0/minicluster-data/master-0/wal/instance:
uuid: "79ce11dccdfe41ac9175b32411225ac3"
format_stamp: "Formatted at 2026-09-26 15:08:46 on dist-test-slave-865v"
server_key: "7900e656bbe5e7e49e75ee966ca64a10"
server_key_iv: "2ad8876a74fc471b9eae8882c9c3ab12"
server_key_version: "encryptionkey@0"
I20260926 15:08:46.053674 12969 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260926 15:08:46.056834 12984 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:08:46.058197 12969 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 15:08:46.058393 12969 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435318278681-12427-0/minicluster-data/master-0/data,/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435318278681-12427-0/minicluster-data/master-0/wal
uuid: "79ce11dccdfe41ac9175b32411225ac3"
format_stamp: "Formatted at 2026-09-26 15:08:46 on dist-test-slave-865v"
server_key: "7900e656bbe5e7e49e75ee966ca64a10"
server_key_iv: "2ad8876a74fc471b9eae8882c9c3ab12"
server_key_version: "encryptionkey@0"
I20260926 15:08:46.058579 12969 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435318278681-12427-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435318278681-12427-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435318278681-12427-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:08:46.075698 12969 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:08:46.079638 12969 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:08:46.079917 12969 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:08:46.099378 12969 rpc_server.cc:307] RPC server started. Bound to: 127.12.34.254:35853
I20260926 15:08:46.099462 13035 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.34.254:35853 every 8 connection(s)
I20260926 15:08:46.100695 12969 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
I20260926 15:08:46.104405 13036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:08:46.108093 12427 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8uoJGv/build/asan/bin/kudu as pid 12969
I20260926 15:08:46.108325 12427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435318278681-12427-0/minicluster-data/master-0/wal/instance
I20260926 15:08:46.109130 12427 external_mini_cluster.cc:1428] Setting key 532acc7c91cfcdceb45fc4bc468c603a
I20260926 15:08:46.117631 13036 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79ce11dccdfe41ac9175b32411225ac3: Bootstrap starting.
I20260926 15:08:46.121304 13036 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 79ce11dccdfe41ac9175b32411225ac3: Neither blocks nor log segments found. Creating new log.
I20260926 15:08:46.122592 13036 log.cc:824] T 00000000000000000000000000000000 P 79ce11dccdfe41ac9175b32411225ac3: Log is configured to *not* fsync() on all Append() calls
Sep 26 15:08:46 dist-test-slave-865v krb5kdc[12956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435325, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.34.254@KRBTEST.COM
I20260926 15:08:46.125499 13036 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79ce11dccdfe41ac9175b32411225ac3: No bootstrap required, opened a new log
I20260926 15:08:46.131615 13039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:08:46.111091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56560 (local address 127.12.34.254:35853)
0926 15:08:46.111800 (+ 709us) server_negotiation.cc:206] Beginning negotiation
0926 15:08:46.111811 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 15:08:46.111853 (+ 42us) server_negotiation.cc:407] Connection header received
0926 15:08:46.113472 (+ 1619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:08:46.113494 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:08:46.113881 (+ 387us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:08:46.114197 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:08:46.115975 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:08:46.117445 (+ 1470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:08:46.119144 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:08:46.119798 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:08:46.119919 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:08:46.124456 (+ 4537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:08:46.124464 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:08:46.124477 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:08:46.124510 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:08:46.127651 (+ 3141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:08:46.128324 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:08:46.128344 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:08:46.128350 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:08:46.128517 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:08:46.128881 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:08:46.128888 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:08:46.128893 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:08:46.129460 (+ 567us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:08:46.129759 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0926 15:08:46.130261 (+ 502us) server_negotiation.cc:299] Negotiation successful
0926 15:08:46.130647 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":513,"thread_start_us":347,"threads_started":1}
I20260926 15:08:46.133256 13036 raft_consensus.cc:348] T 00000000000000000000000000000000 P 79ce11dccdfe41ac9175b32411225ac3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79ce11dccdfe41ac9175b32411225ac3" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 35853 } }
I20260926 15:08:46.133601 13036 raft_consensus.cc:374] T 00000000000000000000000000000000 P 79ce11dccdfe41ac9175b32411225ac3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:08:46.133721 13036 raft_consensus.cc:729] T 00000000000000000000000000000000 P 79ce11dccdfe41ac9175b32411225ac3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79ce11dccdfe41ac9175b32411225ac3, State: Initialized, Role: FOLLOWER
I20260926 15:08:46.134352 13036 consensus_queue.cc:260] T 00000000000000000000000000000000 P 79ce11dccdfe41ac9175b32411225ac3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79ce11dccdfe41ac9175b32411225ac3" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 35853 } }
I20260926 15:08:46.134615 13036 raft_consensus.cc:388] T 00000000000000000000000000000000 P 79ce11dccdfe41ac9175b32411225ac3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 15:08:46.134763 13036 raft_consensus.cc:482] T 00000000000000000000000000000000 P 79ce11dccdfe41ac9175b32411225ac3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 15:08:46.134930 13036 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 79ce11dccdfe41ac9175b32411225ac3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:08:46.137066 13036 raft_consensus.cc:504] T 00000000000000000000000000000000 P 79ce11dccdfe41ac9175b32411225ac3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79ce11dccdfe41ac9175b32411225ac3" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 35853 } }
I20260926 15:08:46.137602 13036 leader_election.cc:302] T 00000000000000000000000000000000 P 79ce11dccdfe41ac9175b32411225ac3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 79ce11dccdfe41ac9175b32411225ac3; no voters:
I20260926 15:08:46.138273 13036 leader_election.cc:288] T 00000000000000000000000000000000 P 79ce11dccdfe41ac9175b32411225ac3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 15:08:46.138453 13041 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 79ce11dccdfe41ac9175b32411225ac3 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:08:46.140208 13036 sys_catalog.cc:564] T 00000000000000000000000000000000 P 79ce11dccdfe41ac9175b32411225ac3 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:08:46.140766 13041 raft_consensus.cc:686] T 00000000000000000000000000000000 P 79ce11dccdfe41ac9175b32411225ac3 [term 1 LEADER]: Becoming Leader. State: Replica: 79ce11dccdfe41ac9175b32411225ac3, State: Running, Role: LEADER
I20260926 15:08:46.141305 13041 consensus_queue.cc:237] T 00000000000000000000000000000000 P 79ce11dccdfe41ac9175b32411225ac3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79ce11dccdfe41ac9175b32411225ac3" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 35853 } }
I20260926 15:08:46.145828 13043 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79ce11dccdfe41ac9175b32411225ac3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 79ce11dccdfe41ac9175b32411225ac3. Latest consensus state: current_term: 1 leader_uuid: "79ce11dccdfe41ac9175b32411225ac3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79ce11dccdfe41ac9175b32411225ac3" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 35853 } } }
I20260926 15:08:46.146175 13043 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79ce11dccdfe41ac9175b32411225ac3 [sys.catalog]: This master's current role is: LEADER
I20260926 15:08:46.146639 13042 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79ce11dccdfe41ac9175b32411225ac3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "79ce11dccdfe41ac9175b32411225ac3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79ce11dccdfe41ac9175b32411225ac3" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 35853 } } }
I20260926 15:08:46.146903 13042 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79ce11dccdfe41ac9175b32411225ac3 [sys.catalog]: This master's current role is: LEADER
I20260926 15:08:46.147917 13055 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:08:46.151962 13055 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 15:08:46.159991 13055 catalog_manager.cc:1347] Generated new cluster ID: 4d40cda5f2b8410bb992cf983f9ce4fe
I20260926 15:08:46.160096 13055 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 15:08:46.177307 13055 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 15:08:46.178514 13055 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:08:46.186252 13055 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 79ce11dccdfe41ac9175b32411225ac3: Generated new TSK 0
I20260926 15:08:46.187279 13055 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.34.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.34.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.34.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.193.keytab.
Entry for principal kudu/127.12.34.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.193.keytab.
I20260926 15:08:46.238616 12427 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8uoJGv/build/asan/bin/kudu
/tmp/dist-test-task8uoJGv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435318278681-12427-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435318278681-12427-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435318278681-12427-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435318278681-12427-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.34.193:0
--local_ip_for_outbound_sockets=127.12.34.193
--webserver_interface=127.12.34.193
--webserver_port=0
--tserver_master_addrs=127.12.34.254:35853
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.193.keytab
--principal=kudu/127.12.34.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.34.212:46693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435318278681-12427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435318278681-12427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435318278681-12427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:08:46.559437 13062 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:08:46.559680 13062 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:08:46.559825 13062 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:08:46.568135 13062 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:08:46.568284 13062 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:08:46.568535 13062 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.34.193
I20260926 15:08:46.579982 13062 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.34.212:46693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435318278681-12427-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435318278681-12427-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.193.keytab
--principal=kudu/127.12.34.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.34.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435318278681-12427-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.34.193
--webserver_port=0
--tserver_master_addrs=127.12.34.254:35853
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.34.193
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435318278681-12427-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:08:46.581172 13062 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:08:46.582401 13062 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:08:46.595297 13067 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:08:46.595322 13068 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:08:46.595327 13070 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:08:46.595996 13062 server_base.cc:1034] running on GCE node
I20260926 15:08:46.596621 13062 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:08:46.597514 13062 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:08:46.598748 13062 hybrid_clock.cc:648] HybridClock initialized: now 1790435326598710 us; error 51 us; skew 500 ppm
I20260926 15:08:46.599047 13062 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:08:46 dist-test-slave-865v krb5kdc[12956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435326, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:08:46.602128 13062 init.cc:377] Logged in from keytab as kudu/127.12.34.193@KRBTEST.COM (short username kudu)
I20260926 15:08:46.603536 13062 webserver.cc:460] Webserver started at http://127.12.34.193:45381/ using document root <none> and password file <none>
I20260926 15:08:46.604131 13062 fs_manager.cc:359] Metadata directory not provided
I20260926 15:08:46.604223 13062 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:08:46.604468 13062 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:08:46.607213 13062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435318278681-12427-0/minicluster-data/ts-0/data/instance:
uuid: "0f54da405e424f52a4f1e4dd91d06fd0"
format_stamp: "Formatted at 2026-09-26 15:08:46 on dist-test-slave-865v"
server_key: "959abd6f3568860066da64848b93f7ec"
server_key_iv: "5d194a755adcb99bbd1dc7752a803268"
server_key_version: "encryptionkey@0"
I20260926 15:08:46.607903 13062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435318278681-12427-0/minicluster-data/ts-0/wal/instance:
uuid: "0f54da405e424f52a4f1e4dd91d06fd0"
format_stamp: "Formatted at 2026-09-26 15:08:46 on dist-test-slave-865v"
server_key: "959abd6f3568860066da64848b93f7ec"
server_key_iv: "5d194a755adcb99bbd1dc7752a803268"
server_key_version: "encryptionkey@0"
I20260926 15:08:46.612573 13062 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260926 15:08:46.615855 13077 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:08:46.617170 13062 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 15:08:46.617328 13062 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435318278681-12427-0/minicluster-data/ts-0/data,/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435318278681-12427-0/minicluster-data/ts-0/wal
uuid: "0f54da405e424f52a4f1e4dd91d06fd0"
format_stamp: "Formatted at 2026-09-26 15:08:46 on dist-test-slave-865v"
server_key: "959abd6f3568860066da64848b93f7ec"
server_key_iv: "5d194a755adcb99bbd1dc7752a803268"
server_key_version: "encryptionkey@0"
I20260926 15:08:46.617518 13062 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435318278681-12427-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435318278681-12427-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435318278681-12427-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:08:46.634831 13062 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:08:46.638656 13062 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:08:46.638923 13062 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:08:46.639811 13062 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:08:46.641161 13062 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:08:46.641232 13062 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:08:46.641363 13062 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:08:46.641424 13062 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:08:46.674444 13062 rpc_server.cc:307] RPC server started. Bound to: 127.12.34.193:44113
I20260926 15:08:46.674482 13189 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.34.193:44113 every 8 connection(s)
I20260926 15:08:46.675642 13062 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435318278681-12427-0/minicluster-data/ts-0/data/info.pb
I20260926 15:08:46.685643 12427 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8uoJGv/build/asan/bin/kudu as pid 13062
I20260926 15:08:46.685878 12427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435318278681-12427-0/minicluster-data/ts-0/wal/instance
I20260926 15:08:46.686671 12427 external_mini_cluster.cc:1428] Setting key bfb097451f42ac2a4cf04eaea1b9ddc6
Sep 26 15:08:46 dist-test-slave-865v krb5kdc[12956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435326, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.193@KRBTEST.COM for kudu/127.12.34.254@KRBTEST.COM
I20260926 15:08:46.695165 13192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:08:46.678253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.193:49497 (local address 127.12.34.254:35853)
0926 15:08:46.678673 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0926 15:08:46.678678 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:08:46.679666 (+ 988us) server_negotiation.cc:407] Connection header received
0926 15:08:46.681471 (+ 1805us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:08:46.681475 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:08:46.681555 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:08:46.681648 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:08:46.683112 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:08:46.683795 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:08:46.685322 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:08:46.685855 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:08:46.686014 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:08:46.689818 (+ 3804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:08:46.689823 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:08:46.689826 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:08:46.689855 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:08:46.692249 (+ 2394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:08:46.692947 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:08:46.692952 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:08:46.692958 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:08:46.693117 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:08:46.693541 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:08:46.693546 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:08:46.693550 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:08:46.693821 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:08:46.693974 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0926 15:08:46.694607 (+ 633us) server_negotiation.cc:299] Negotiation successful
0926 15:08:46.694806 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":215,"threads_started":1}
I20260926 15:08:46.696070 13193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:08:46.678842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:35853 (local address 127.12.34.193:49497)
0926 15:08:46.679499 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0926 15:08:46.679543 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 15:08:46.681234 (+ 1691us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:08:46.681907 (+ 673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:08:46.681915 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:08:46.682400 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:08:46.682971 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:08:46.682988 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:08:46.683932 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:08:46.683935 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:08:46.685174 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:08:46.685186 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:08:46.686138 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:08:46.686148 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:08:46.686473 (+ 325us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:08:46.687174 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:08:46.687202 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:08:46.689605 (+ 2403us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:08:46.692388 (+ 2783us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:08:46.692397 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:08:46.692413 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:08:46.692821 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:08:46.693233 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:08:46.693238 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:08:46.693242 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:08:46.693426 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:08:46.693950 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:08:46.693958 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:08:46.694274 (+ 316us) client_negotiation.cc:769] Sending connection context
0926 15:08:46.694529 (+ 255us) client_negotiation.cc:240] Negotiation successful
0926 15:08:46.695032 (+ 503us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":258,"threads_started":1}
I20260926 15:08:46.697557 13190 heartbeater.cc:347] Connected to a master server at 127.12.34.254:35853
I20260926 15:08:46.697865 13190 heartbeater.cc:464] Registering TS with master...
I20260926 15:08:46.698757 13190 heartbeater.cc:511] Master 127.12.34.254:35853 requested a full tablet report, sending...
I20260926 15:08:46.700953 13001 ts_manager.cc:194] Registered new tserver with Master: 0f54da405e424f52a4f1e4dd91d06fd0 (127.12.34.193:44113)
WARNING: no policy specified for kudu/127.12.34.194@KRBTEST.COM; defaulting to no policy
I20260926 15:08:46.702486 13001 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.34.193@KRBTEST.COM'} at 127.12.34.193:49497
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.34.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.34.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.194.keytab.
Entry for principal kudu/127.12.34.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.194.keytab.
I20260926 15:08:46.724820 12427 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8uoJGv/build/asan/bin/kudu
/tmp/dist-test-task8uoJGv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435318278681-12427-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435318278681-12427-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435318278681-12427-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435318278681-12427-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.34.194:0
--local_ip_for_outbound_sockets=127.12.34.194
--webserver_interface=127.12.34.194
--webserver_port=0
--tserver_master_addrs=127.12.34.254:35853
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.194.keytab
--principal=kudu/127.12.34.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.34.212:46693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435318278681-12427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435318278681-12427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435318278681-12427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:08:47.036928 13196 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:08:47.037292 13196 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:08:47.037469 13196 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:08:47.045472 13196 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:08:47.045629 13196 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:08:47.045804 13196 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.34.194
I20260926 15:08:47.056222 13196 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.34.212:46693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435318278681-12427-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435318278681-12427-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.194.keytab
--principal=kudu/127.12.34.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.34.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435318278681-12427-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.34.194
--webserver_port=0
--tserver_master_addrs=127.12.34.254:35853
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.34.194
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435318278681-12427-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:08:47.057641 13196 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:08:47.058998 13196 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:08:47.072538 13201 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:08:47.072549 13204 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:08:47.072507 13202 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:08:47.073149 13196 server_base.cc:1034] running on GCE node
I20260926 15:08:47.074124 13196 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:08:47.075326 13196 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:08:47.076582 13196 hybrid_clock.cc:648] HybridClock initialized: now 1790435327076542 us; error 66 us; skew 500 ppm
I20260926 15:08:47.076913 13196 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:08:47 dist-test-slave-865v krb5kdc[12956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435327, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:08:47.080042 13196 init.cc:377] Logged in from keytab as kudu/127.12.34.194@KRBTEST.COM (short username kudu)
I20260926 15:08:47.081741 13196 webserver.cc:460] Webserver started at http://127.12.34.194:34049/ using document root <none> and password file <none>
I20260926 15:08:47.082418 13196 fs_manager.cc:359] Metadata directory not provided
I20260926 15:08:47.082563 13196 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:08:47.082876 13196 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:08:47.085783 13196 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435318278681-12427-0/minicluster-data/ts-1/data/instance:
uuid: "872a2f2a4bf1419b89b7dbdd81ff8745"
format_stamp: "Formatted at 2026-09-26 15:08:47 on dist-test-slave-865v"
server_key: "2fcf189246da873045339b39808a79ef"
server_key_iv: "f3ffcfbf065bbaee88022948e8efca24"
server_key_version: "encryptionkey@0"
I20260926 15:08:47.086552 13196 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435318278681-12427-0/minicluster-data/ts-1/wal/instance:
uuid: "872a2f2a4bf1419b89b7dbdd81ff8745"
format_stamp: "Formatted at 2026-09-26 15:08:47 on dist-test-slave-865v"
server_key: "2fcf189246da873045339b39808a79ef"
server_key_iv: "f3ffcfbf065bbaee88022948e8efca24"
server_key_version: "encryptionkey@0"
I20260926 15:08:47.091902 13196 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 15:08:47.095335 13211 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:08:47.096616 13196 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260926 15:08:47.096796 13196 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435318278681-12427-0/minicluster-data/ts-1/data,/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435318278681-12427-0/minicluster-data/ts-1/wal
uuid: "872a2f2a4bf1419b89b7dbdd81ff8745"
format_stamp: "Formatted at 2026-09-26 15:08:47 on dist-test-slave-865v"
server_key: "2fcf189246da873045339b39808a79ef"
server_key_iv: "f3ffcfbf065bbaee88022948e8efca24"
server_key_version: "encryptionkey@0"
I20260926 15:08:47.096993 13196 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435318278681-12427-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435318278681-12427-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435318278681-12427-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:08:47.122112 13196 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:08:47.126320 13196 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:08:47.126607 13196 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:08:47.127573 13196 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:08:47.128841 13196 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:08:47.128948 13196 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:08:47.129150 13196 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:08:47.129222 13196 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260926 15:08:47.163959 13196 rpc_server.cc:307] RPC server started. Bound to: 127.12.34.194:36713
I20260926 15:08:47.164017 13323 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.34.194:36713 every 8 connection(s)
I20260926 15:08:47.165261 13196 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435318278681-12427-0/minicluster-data/ts-1/data/info.pb
I20260926 15:08:47.171571 12427 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8uoJGv/build/asan/bin/kudu as pid 13196
I20260926 15:08:47.171777 12427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435318278681-12427-0/minicluster-data/ts-1/wal/instance
I20260926 15:08:47.172207 12427 external_mini_cluster.cc:1428] Setting key 05e532b86cf0ad1a6f19b113aaa053c5
Sep 26 15:08:47 dist-test-slave-865v krb5kdc[12956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435327, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.194@KRBTEST.COM for kudu/127.12.34.254@KRBTEST.COM
I20260926 15:08:47.185356 13192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:08:47.167790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.194:39119 (local address 127.12.34.254:35853)
0926 15:08:47.167991 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0926 15:08:47.167995 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:08:47.169238 (+ 1243us) server_negotiation.cc:407] Connection header received
0926 15:08:47.171014 (+ 1776us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:08:47.171020 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:08:47.171117 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:08:47.171257 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:08:47.172889 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:08:47.173805 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:08:47.175377 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:08:47.176138 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:08:47.176284 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:08:47.179505 (+ 3221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:08:47.179510 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:08:47.179513 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:08:47.179542 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:08:47.182126 (+ 2584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:08:47.183070 (+ 944us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:08:47.183075 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:08:47.183077 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:08:47.183203 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:08:47.183677 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:08:47.183680 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:08:47.183682 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:08:47.183919 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:08:47.184041 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0926 15:08:47.184895 (+ 854us) server_negotiation.cc:299] Negotiation successful
0926 15:08:47.185067 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260926 15:08:47.187007 13326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:08:47.168303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:35853 (local address 127.12.34.194:39119)
0926 15:08:47.169019 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0926 15:08:47.169063 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 15:08:47.170760 (+ 1697us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:08:47.171469 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:08:47.171478 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:08:47.172100 (+ 622us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:08:47.172701 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:08:47.172727 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:08:47.173956 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:08:47.173961 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:08:47.175229 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:08:47.175240 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:08:47.176332 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:08:47.176337 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:08:47.176510 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:08:47.177192 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:08:47.177213 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:08:47.179311 (+ 2098us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:08:47.182267 (+ 2956us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:08:47.182278 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:08:47.182298 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:08:47.182934 (+ 636us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:08:47.183321 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:08:47.183329 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:08:47.183336 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:08:47.183560 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:08:47.184051 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:08:47.184063 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:08:47.184500 (+ 437us) client_negotiation.cc:769] Sending connection context
0926 15:08:47.184825 (+ 325us) client_negotiation.cc:240] Negotiation successful
0926 15:08:47.185426 (+ 601us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":305,"threads_started":1}
I20260926 15:08:47.188495 13324 heartbeater.cc:347] Connected to a master server at 127.12.34.254:35853
WARNING: no policy specified for kudu/127.12.34.195@KRBTEST.COM; defaulting to no policy
I20260926 15:08:47.188854 13324 heartbeater.cc:464] Registering TS with master...
I20260926 15:08:47.189630 13324 heartbeater.cc:511] Master 127.12.34.254:35853 requested a full tablet report, sending...
I20260926 15:08:47.191184 13001 ts_manager.cc:194] Registered new tserver with Master: 872a2f2a4bf1419b89b7dbdd81ff8745 (127.12.34.194:36713)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.34.195@KRBTEST.COM" created.
I20260926 15:08:47.192267 13001 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.34.194@KRBTEST.COM'} at 127.12.34.194:39119
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.34.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.195.keytab.
Entry for principal kudu/127.12.34.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.195.keytab.
I20260926 15:08:47.212834 12427 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8uoJGv/build/asan/bin/kudu
/tmp/dist-test-task8uoJGv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435318278681-12427-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435318278681-12427-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435318278681-12427-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435318278681-12427-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.34.195:0
--local_ip_for_outbound_sockets=127.12.34.195
--webserver_interface=127.12.34.195
--webserver_port=0
--tserver_master_addrs=127.12.34.254:35853
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.195.keytab
--principal=kudu/127.12.34.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.34.212:46693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435318278681-12427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435318278681-12427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435318278681-12427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:08:47.526775 13329 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:08:47.527050 13329 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:08:47.527145 13329 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:08:47.534219 13329 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:08:47.534327 13329 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:08:47.534509 13329 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.34.195
I20260926 15:08:47.544245 13329 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.34.212:46693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435318278681-12427-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435318278681-12427-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.195.keytab
--principal=kudu/127.12.34.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.34.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435318278681-12427-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.34.195
--webserver_port=0
--tserver_master_addrs=127.12.34.254:35853
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.34.195
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435318278681-12427-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:08:47.545559 13329 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:08:47.546837 13329 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:08:47.559438 13334 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:08:47.559490 13337 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:08:47.559525 13335 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:08:47.560304 13329 server_base.cc:1034] running on GCE node
I20260926 15:08:47.560873 13329 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:08:47.561911 13329 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:08:47.563146 13329 hybrid_clock.cc:648] HybridClock initialized: now 1790435327563102 us; error 39 us; skew 500 ppm
I20260926 15:08:47.563469 13329 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:08:47 dist-test-slave-865v krb5kdc[12956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435327, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:08:47.566620 13329 init.cc:377] Logged in from keytab as kudu/127.12.34.195@KRBTEST.COM (short username kudu)
I20260926 15:08:47.568005 13329 webserver.cc:460] Webserver started at http://127.12.34.195:46333/ using document root <none> and password file <none>
I20260926 15:08:47.568593 13329 fs_manager.cc:359] Metadata directory not provided
I20260926 15:08:47.568687 13329 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:08:47.568917 13329 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:08:47.571631 13329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435318278681-12427-0/minicluster-data/ts-2/data/instance:
uuid: "ed1a343d43004d34b76cdab47959592f"
format_stamp: "Formatted at 2026-09-26 15:08:47 on dist-test-slave-865v"
server_key: "fc6c714f64d584596e52a55105c58826"
server_key_iv: "6d8621f8fde968c4fa725584908fae08"
server_key_version: "encryptionkey@0"
I20260926 15:08:47.572319 13329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435318278681-12427-0/minicluster-data/ts-2/wal/instance:
uuid: "ed1a343d43004d34b76cdab47959592f"
format_stamp: "Formatted at 2026-09-26 15:08:47 on dist-test-slave-865v"
server_key: "fc6c714f64d584596e52a55105c58826"
server_key_iv: "6d8621f8fde968c4fa725584908fae08"
server_key_version: "encryptionkey@0"
I20260926 15:08:47.577035 13329 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.002s
I20260926 15:08:47.580415 13344 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:08:47.581949 13329 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 15:08:47.582119 13329 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435318278681-12427-0/minicluster-data/ts-2/data,/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435318278681-12427-0/minicluster-data/ts-2/wal
uuid: "ed1a343d43004d34b76cdab47959592f"
format_stamp: "Formatted at 2026-09-26 15:08:47 on dist-test-slave-865v"
server_key: "fc6c714f64d584596e52a55105c58826"
server_key_iv: "6d8621f8fde968c4fa725584908fae08"
server_key_version: "encryptionkey@0"
I20260926 15:08:47.582307 13329 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435318278681-12427-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435318278681-12427-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435318278681-12427-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:08:47.623744 13329 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:08:47.628064 13329 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:08:47.628353 13329 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:08:47.629312 13329 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:08:47.630587 13329 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:08:47.630692 13329 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:08:47.630832 13329 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:08:47.630903 13329 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:08:47.663884 13329 rpc_server.cc:307] RPC server started. Bound to: 127.12.34.195:35591
I20260926 15:08:47.663934 13456 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.34.195:35591 every 8 connection(s)
I20260926 15:08:47.665254 13329 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435318278681-12427-0/minicluster-data/ts-2/data/info.pb
I20260926 15:08:47.668882 12427 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8uoJGv/build/asan/bin/kudu as pid 13329
I20260926 15:08:47.669072 12427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790435318278681-12427-0/minicluster-data/ts-2/wal/instance
I20260926 15:08:47.669597 12427 external_mini_cluster.cc:1428] Setting key d6465b654effae7344788f7b2fefa20c
Sep 26 15:08:47 dist-test-slave-865v krb5kdc[12956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435327, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.195@KRBTEST.COM for kudu/127.12.34.254@KRBTEST.COM
I20260926 15:08:47.687094 13192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:08:47.667916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.195:49097 (local address 127.12.34.254:35853)
0926 15:08:47.668130 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0926 15:08:47.668135 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:08:47.669406 (+ 1271us) server_negotiation.cc:407] Connection header received
0926 15:08:47.671714 (+ 2308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:08:47.671719 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:08:47.671798 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:08:47.671927 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:08:47.673796 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:08:47.674918 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:08:47.676476 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:08:47.677129 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:08:47.677253 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:08:47.680618 (+ 3365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:08:47.680623 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:08:47.680626 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:08:47.680657 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:08:47.683433 (+ 2776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:08:47.684277 (+ 844us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:08:47.684283 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:08:47.684287 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:08:47.684472 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:08:47.684982 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:08:47.685010 (+ 28us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:08:47.685016 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:08:47.685492 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:08:47.685624 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0926 15:08:47.686563 (+ 939us) server_negotiation.cc:299] Negotiation successful
0926 15:08:47.686724 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260926 15:08:47.688102 13459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:08:47.668381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:35853 (local address 127.12.34.195:49097)
0926 15:08:47.669259 (+ 878us) negotiation.cc:107] Waiting for socket to connect
0926 15:08:47.669296 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 15:08:47.671337 (+ 2041us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:08:47.672185 (+ 848us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:08:47.672206 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:08:47.673014 (+ 808us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:08:47.673634 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:08:47.673651 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:08:47.675095 (+ 1444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:08:47.675100 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:08:47.676330 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:08:47.676342 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:08:47.677333 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:08:47.677338 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:08:47.677620 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:08:47.678308 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:08:47.678331 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:08:47.680400 (+ 2069us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:08:47.683618 (+ 3218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:08:47.683632 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:08:47.683644 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:08:47.684131 (+ 487us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:08:47.684597 (+ 466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:08:47.684602 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:08:47.684605 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:08:47.684855 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:08:47.685676 (+ 821us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:08:47.685685 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:08:47.686174 (+ 489us) client_negotiation.cc:769] Sending connection context
0926 15:08:47.686445 (+ 271us) client_negotiation.cc:240] Negotiation successful
0926 15:08:47.686883 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":329,"threads_started":1}
I20260926 15:08:47.689535 13457 heartbeater.cc:347] Connected to a master server at 127.12.34.254:35853
I20260926 15:08:47.689827 13457 heartbeater.cc:464] Registering TS with master...
I20260926 15:08:47.690546 13457 heartbeater.cc:511] Master 127.12.34.254:35853 requested a full tablet report, sending...
I20260926 15:08:47.692368 13001 ts_manager.cc:194] Registered new tserver with Master: ed1a343d43004d34b76cdab47959592f (127.12.34.195:35591)
I20260926 15:08:47.693244 13001 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.34.195@KRBTEST.COM'} at 127.12.34.195:49097
I20260926 15:08:47.698254 12427 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 15:08:47.705852 13190 heartbeater.cc:503] Master 127.12.34.254:35853 was elected leader, sending a full tablet report...
I20260926 15:08:47.717350 13192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:08:47.704389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56572 (local address 127.12.34.254:35853)
0926 15:08:47.704568 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0926 15:08:47.704572 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:08:47.705272 (+ 700us) server_negotiation.cc:407] Connection header received
0926 15:08:47.705406 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:08:47.705413 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:08:47.705537 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:08:47.705811 (+ 274us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:08:47.707551 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:08:47.708291 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:08:47.709638 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:08:47.710042 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:08:47.710178 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:08:47.712394 (+ 2216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:08:47.712399 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:08:47.712403 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:08:47.712431 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:08:47.714776 (+ 2345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:08:47.715390 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:08:47.715401 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:08:47.715403 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:08:47.715527 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:08:47.715892 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:08:47.715899 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:08:47.715903 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:08:47.716210 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:08:47.716457 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0926 15:08:47.716791 (+ 334us) server_negotiation.cc:299] Negotiation successful
0926 15:08:47.717000 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260926 15:08:47.724915 13001 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56572:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 15:08:47.731765 13001 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 15:08:47.760344 13469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:08:47.746762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.193:44113 (local address 127.0.0.1:43494)
0926 15:08:47.747773 (+ 1011us) negotiation.cc:107] Waiting for socket to connect
0926 15:08:47.747836 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0926 15:08:47.748051 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:08:47.749518 (+ 1467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:08:47.749526 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:08:47.749561 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:08:47.749971 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:08:47.749996 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:08:47.752804 (+ 2808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:08:47.752813 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:08:47.755908 (+ 3095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:08:47.755929 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:08:47.758523 (+ 2594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:08:47.758532 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:08:47.758872 (+ 340us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:08:47.758951 (+ 79us) client_negotiation.cc:769] Sending connection context
0926 15:08:47.759090 (+ 139us) client_negotiation.cc:240] Negotiation successful
0926 15:08:47.759355 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":665,"thread_start_us":456,"threads_started":1}
I20260926 15:08:47.762444 13470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:08:47.747062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43494 (local address 127.12.34.193:44113)
0926 15:08:47.748778 (+ 1716us) server_negotiation.cc:206] Beginning negotiation
0926 15:08:47.748788 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 15:08:47.748819 (+ 31us) server_negotiation.cc:407] Connection header received
0926 15:08:47.748911 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:08:47.748920 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:08:47.749253 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:08:47.750217 (+ 964us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:08:47.750512 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:08:47.752615 (+ 2103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:08:47.756178 (+ 3563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:08:47.758217 (+ 2039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:08:47.761206 (+ 2989us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:08:47.761543 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0926 15:08:47.761632 (+ 89us) server_negotiation.cc:299] Negotiation successful
0926 15:08:47.761914 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1544,"thread_start_us":1376,"threads_started":1}
I20260926 15:08:47.763753 13471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:08:47.747102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.194:36713 (local address 127.0.0.1:60434)
0926 15:08:47.749216 (+ 2114us) negotiation.cc:107] Waiting for socket to connect
0926 15:08:47.749233 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 15:08:47.749343 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:08:47.750382 (+ 1039us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:08:47.750389 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:08:47.750422 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:08:47.752270 (+ 1848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:08:47.752288 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:08:47.754470 (+ 2182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:08:47.754479 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:08:47.758720 (+ 4241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:08:47.758741 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:08:47.762974 (+ 4233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:08:47.762980 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:08:47.763177 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:08:47.763200 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 15:08:47.763305 (+ 105us) client_negotiation.cc:240] Negotiation successful
0926 15:08:47.763433 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1982,"thread_start_us":1820,"threads_started":1}
I20260926 15:08:47.765093 13472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:08:47.747660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60434 (local address 127.12.34.194:36713)
0926 15:08:47.748991 (+ 1331us) server_negotiation.cc:206] Beginning negotiation
0926 15:08:47.748999 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 15:08:47.749474 (+ 475us) server_negotiation.cc:407] Connection header received
0926 15:08:47.749590 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:08:47.749601 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:08:47.749863 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:08:47.750088 (+ 225us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:08:47.752497 (+ 2409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:08:47.754273 (+ 1776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:08:47.760254 (+ 5981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:08:47.762270 (+ 2016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:08:47.762444 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:08:47.762702 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0926 15:08:47.763361 (+ 659us) server_negotiation.cc:299] Negotiation successful
0926 15:08:47.763541 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1187,"thread_start_us":1085,"threads_started":1}
I20260926 15:08:47.766786 13125 tablet_service.cc:1467] Processing CreateTablet for tablet de3763cd55b04f0088c4b8f68d157ff2 (DEFAULT_TABLE table=test-table [id=7e159392818941cba815fc3cbdecdcae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:08:47.768019 13125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de3763cd55b04f0088c4b8f68d157ff2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:08:47.768376 13259 tablet_service.cc:1467] Processing CreateTablet for tablet de3763cd55b04f0088c4b8f68d157ff2 (DEFAULT_TABLE table=test-table [id=7e159392818941cba815fc3cbdecdcae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:08:47.769717 13259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de3763cd55b04f0088c4b8f68d157ff2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:08:47.770622 13469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:08:47.749323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.195:35591 (local address 127.0.0.1:35520)
0926 15:08:47.760616 (+ 11293us) negotiation.cc:107] Waiting for socket to connect
0926 15:08:47.760634 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 15:08:47.760723 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:08:47.764076 (+ 3353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:08:47.764081 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:08:47.764105 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:08:47.764274 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:08:47.764284 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:08:47.766344 (+ 2060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:08:47.766349 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:08:47.768101 (+ 1752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:08:47.768115 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:08:47.769907 (+ 1792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:08:47.769913 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:08:47.770094 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:08:47.770127 (+ 33us) client_negotiation.cc:769] Sending connection context
0926 15:08:47.770196 (+ 69us) client_negotiation.cc:240] Negotiation successful
0926 15:08:47.770333 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11192,"thread_start_us":13605,"threads_started":1}
I20260926 15:08:47.771814 13474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:08:47.749661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35520 (local address 127.12.34.195:35591)
0926 15:08:47.763515 (+ 13854us) server_negotiation.cc:206] Beginning negotiation
0926 15:08:47.763525 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 15:08:47.763559 (+ 34us) server_negotiation.cc:407] Connection header received
0926 15:08:47.763658 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:08:47.763665 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:08:47.763910 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:08:47.764095 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:08:47.764425 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:08:47.766179 (+ 1754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:08:47.768278 (+ 2099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:08:47.769727 (+ 1449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:08:47.770987 (+ 1260us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:08:47.771213 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0926 15:08:47.771280 (+ 67us) server_negotiation.cc:299] Negotiation successful
0926 15:08:47.771458 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13669,"thread_start_us":13985,"threads_started":1}
I20260926 15:08:47.775426 13392 tablet_service.cc:1467] Processing CreateTablet for tablet de3763cd55b04f0088c4b8f68d157ff2 (DEFAULT_TABLE table=test-table [id=7e159392818941cba815fc3cbdecdcae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:08:47.776777 13392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de3763cd55b04f0088c4b8f68d157ff2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:08:47.780769 13475 tablet_bootstrap.cc:492] T de3763cd55b04f0088c4b8f68d157ff2 P 872a2f2a4bf1419b89b7dbdd81ff8745: Bootstrap starting.
I20260926 15:08:47.783806 13475 tablet_bootstrap.cc:654] T de3763cd55b04f0088c4b8f68d157ff2 P 872a2f2a4bf1419b89b7dbdd81ff8745: Neither blocks nor log segments found. Creating new log.
I20260926 15:08:47.784752 13475 log.cc:824] T de3763cd55b04f0088c4b8f68d157ff2 P 872a2f2a4bf1419b89b7dbdd81ff8745: Log is configured to *not* fsync() on all Append() calls
I20260926 15:08:47.785694 13476 tablet_bootstrap.cc:492] T de3763cd55b04f0088c4b8f68d157ff2 P 0f54da405e424f52a4f1e4dd91d06fd0: Bootstrap starting.
I20260926 15:08:47.789204 13475 tablet_bootstrap.cc:492] T de3763cd55b04f0088c4b8f68d157ff2 P 872a2f2a4bf1419b89b7dbdd81ff8745: No bootstrap required, opened a new log
I20260926 15:08:47.789388 13476 tablet_bootstrap.cc:654] T de3763cd55b04f0088c4b8f68d157ff2 P 0f54da405e424f52a4f1e4dd91d06fd0: Neither blocks nor log segments found. Creating new log.
I20260926 15:08:47.789872 13475 ts_tablet_manager.cc:1397] T de3763cd55b04f0088c4b8f68d157ff2 P 872a2f2a4bf1419b89b7dbdd81ff8745: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.003s
I20260926 15:08:47.790540 13476 log.cc:824] T de3763cd55b04f0088c4b8f68d157ff2 P 0f54da405e424f52a4f1e4dd91d06fd0: Log is configured to *not* fsync() on all Append() calls
I20260926 15:08:47.792904 13479 tablet_bootstrap.cc:492] T de3763cd55b04f0088c4b8f68d157ff2 P ed1a343d43004d34b76cdab47959592f: Bootstrap starting.
I20260926 15:08:47.794032 13476 tablet_bootstrap.cc:492] T de3763cd55b04f0088c4b8f68d157ff2 P 0f54da405e424f52a4f1e4dd91d06fd0: No bootstrap required, opened a new log
I20260926 15:08:47.794286 13476 ts_tablet_manager.cc:1397] T de3763cd55b04f0088c4b8f68d157ff2 P 0f54da405e424f52a4f1e4dd91d06fd0: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260926 15:08:47.795634 13479 tablet_bootstrap.cc:654] T de3763cd55b04f0088c4b8f68d157ff2 P ed1a343d43004d34b76cdab47959592f: Neither blocks nor log segments found. Creating new log.
I20260926 15:08:47.796859 13479 log.cc:824] T de3763cd55b04f0088c4b8f68d157ff2 P ed1a343d43004d34b76cdab47959592f: Log is configured to *not* fsync() on all Append() calls
I20260926 15:08:47.800266 13479 tablet_bootstrap.cc:492] T de3763cd55b04f0088c4b8f68d157ff2 P ed1a343d43004d34b76cdab47959592f: No bootstrap required, opened a new log
I20260926 15:08:47.800083 13475 raft_consensus.cc:348] T de3763cd55b04f0088c4b8f68d157ff2 P 872a2f2a4bf1419b89b7dbdd81ff8745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f54da405e424f52a4f1e4dd91d06fd0" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 44113 } } peers { permanent_uuid: "872a2f2a4bf1419b89b7dbdd81ff8745" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 36713 } } peers { permanent_uuid: "ed1a343d43004d34b76cdab47959592f" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 35591 } }
I20260926 15:08:47.800515 13475 raft_consensus.cc:374] T de3763cd55b04f0088c4b8f68d157ff2 P 872a2f2a4bf1419b89b7dbdd81ff8745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:08:47.800549 13479 ts_tablet_manager.cc:1397] T de3763cd55b04f0088c4b8f68d157ff2 P ed1a343d43004d34b76cdab47959592f: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260926 15:08:47.800606 13475 raft_consensus.cc:729] T de3763cd55b04f0088c4b8f68d157ff2 P 872a2f2a4bf1419b89b7dbdd81ff8745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 872a2f2a4bf1419b89b7dbdd81ff8745, State: Initialized, Role: FOLLOWER
I20260926 15:08:47.801220 13475 consensus_queue.cc:260] T de3763cd55b04f0088c4b8f68d157ff2 P 872a2f2a4bf1419b89b7dbdd81ff8745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f54da405e424f52a4f1e4dd91d06fd0" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 44113 } } peers { permanent_uuid: "872a2f2a4bf1419b89b7dbdd81ff8745" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 36713 } } peers { permanent_uuid: "ed1a343d43004d34b76cdab47959592f" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 35591 } }
I20260926 15:08:47.802309 13324 heartbeater.cc:503] Master 127.12.34.254:35853 was elected leader, sending a full tablet report...
I20260926 15:08:47.802946 13475 ts_tablet_manager.cc:1428] T de3763cd55b04f0088c4b8f68d157ff2 P 872a2f2a4bf1419b89b7dbdd81ff8745: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260926 15:08:47.805371 13476 raft_consensus.cc:348] T de3763cd55b04f0088c4b8f68d157ff2 P 0f54da405e424f52a4f1e4dd91d06fd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f54da405e424f52a4f1e4dd91d06fd0" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 44113 } } peers { permanent_uuid: "872a2f2a4bf1419b89b7dbdd81ff8745" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 36713 } } peers { permanent_uuid: "ed1a343d43004d34b76cdab47959592f" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 35591 } }
I20260926 15:08:47.805712 13476 raft_consensus.cc:374] T de3763cd55b04f0088c4b8f68d157ff2 P 0f54da405e424f52a4f1e4dd91d06fd0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:08:47.805823 13476 raft_consensus.cc:729] T de3763cd55b04f0088c4b8f68d157ff2 P 0f54da405e424f52a4f1e4dd91d06fd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f54da405e424f52a4f1e4dd91d06fd0, State: Initialized, Role: FOLLOWER
I20260926 15:08:47.806375 13476 consensus_queue.cc:260] T de3763cd55b04f0088c4b8f68d157ff2 P 0f54da405e424f52a4f1e4dd91d06fd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f54da405e424f52a4f1e4dd91d06fd0" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 44113 } } peers { permanent_uuid: "872a2f2a4bf1419b89b7dbdd81ff8745" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 36713 } } peers { permanent_uuid: "ed1a343d43004d34b76cdab47959592f" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 35591 } }
I20260926 15:08:47.808110 13476 ts_tablet_manager.cc:1428] T de3763cd55b04f0088c4b8f68d157ff2 P 0f54da405e424f52a4f1e4dd91d06fd0: Time spent starting tablet: real 0.014s user 0.012s sys 0.000s
I20260926 15:08:47.811228 13479 raft_consensus.cc:348] T de3763cd55b04f0088c4b8f68d157ff2 P ed1a343d43004d34b76cdab47959592f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f54da405e424f52a4f1e4dd91d06fd0" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 44113 } } peers { permanent_uuid: "872a2f2a4bf1419b89b7dbdd81ff8745" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 36713 } } peers { permanent_uuid: "ed1a343d43004d34b76cdab47959592f" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 35591 } }
I20260926 15:08:47.811664 13479 raft_consensus.cc:374] T de3763cd55b04f0088c4b8f68d157ff2 P ed1a343d43004d34b76cdab47959592f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:08:47.811762 13479 raft_consensus.cc:729] T de3763cd55b04f0088c4b8f68d157ff2 P ed1a343d43004d34b76cdab47959592f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed1a343d43004d34b76cdab47959592f, State: Initialized, Role: FOLLOWER
I20260926 15:08:47.812288 13479 consensus_queue.cc:260] T de3763cd55b04f0088c4b8f68d157ff2 P ed1a343d43004d34b76cdab47959592f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f54da405e424f52a4f1e4dd91d06fd0" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 44113 } } peers { permanent_uuid: "872a2f2a4bf1419b89b7dbdd81ff8745" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 36713 } } peers { permanent_uuid: "ed1a343d43004d34b76cdab47959592f" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 35591 } }
I20260926 15:08:47.813237 13457 heartbeater.cc:503] Master 127.12.34.254:35853 was elected leader, sending a full tablet report...
I20260926 15:08:47.813812 13479 ts_tablet_manager.cc:1428] T de3763cd55b04f0088c4b8f68d157ff2 P ed1a343d43004d34b76cdab47959592f: Time spent starting tablet: real 0.013s user 0.007s sys 0.007s
I20260926 15:08:47.915017 13482 raft_consensus.cc:482] T de3763cd55b04f0088c4b8f68d157ff2 P 0f54da405e424f52a4f1e4dd91d06fd0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 15:08:47.915313 13482 raft_consensus.cc:504] T de3763cd55b04f0088c4b8f68d157ff2 P 0f54da405e424f52a4f1e4dd91d06fd0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f54da405e424f52a4f1e4dd91d06fd0" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 44113 } } peers { permanent_uuid: "872a2f2a4bf1419b89b7dbdd81ff8745" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 36713 } } peers { permanent_uuid: "ed1a343d43004d34b76cdab47959592f" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 35591 } }
W20260926 15:08:47.916843 13458 tablet.cc:2369] T de3763cd55b04f0088c4b8f68d157ff2 P ed1a343d43004d34b76cdab47959592f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 15:08:47.917241 13325 tablet.cc:2369] T de3763cd55b04f0088c4b8f68d157ff2 P 872a2f2a4bf1419b89b7dbdd81ff8745: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 15:08:47.917654 13482 leader_election.cc:288] T de3763cd55b04f0088c4b8f68d157ff2 P 0f54da405e424f52a4f1e4dd91d06fd0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 872a2f2a4bf1419b89b7dbdd81ff8745 (127.12.34.194:36713), ed1a343d43004d34b76cdab47959592f (127.12.34.195:35591)
I20260926 15:08:47.924227 13472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:08:47.918080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.193:55195 (local address 127.12.34.194:36713)
0926 15:08:47.918301 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0926 15:08:47.918309 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 15:08:47.918877 (+ 568us) server_negotiation.cc:407] Connection header received
0926 15:08:47.919117 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:08:47.919126 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:08:47.919218 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:08:47.919319 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:08:47.919920 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:08:47.920702 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:08:47.922608 (+ 1906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:08:47.923348 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:08:47.923518 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:08:47.923603 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 15:08:47.923868 (+ 265us) server_negotiation.cc:299] Negotiation successful
0926 15:08:47.923987 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 15:08:47.924386 13484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:08:47.918091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.194:36713 (local address 127.12.34.193:55195)
0926 15:08:47.918731 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0926 15:08:47.918747 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 15:08:47.918936 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:08:47.919360 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:08:47.919364 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:08:47.919388 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:08:47.919781 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:08:47.919792 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:08:47.920823 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:08:47.920830 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:08:47.922452 (+ 1622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:08:47.922463 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:08:47.923543 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:08:47.923549 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:08:47.923716 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:08:47.923738 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 15:08:47.923845 (+ 107us) client_negotiation.cc:240] Negotiation successful
0926 15:08:47.923971 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"mutex_wait_us":4,"thread_start_us":332,"threads_started":1}
I20260926 15:08:47.924618 13474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:08:47.918973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.193:44879 (local address 127.12.34.195:35591)
0926 15:08:47.919198 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0926 15:08:47.919206 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 15:08:47.919619 (+ 413us) server_negotiation.cc:407] Connection header received
0926 15:08:47.919782 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:08:47.919786 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:08:47.919840 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:08:47.919941 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:08:47.920464 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:08:47.921253 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:08:47.922989 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:08:47.923718 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:08:47.923894 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:08:47.923964 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 15:08:47.924244 (+ 280us) server_negotiation.cc:299] Negotiation successful
0926 15:08:47.924363 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 15:08:47.924836 13485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:08:47.918874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.195:35591 (local address 127.12.34.193:44879)
0926 15:08:47.919512 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0926 15:08:47.919533 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 15:08:47.919665 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:08:47.919957 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:08:47.919961 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:08:47.919985 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:08:47.920312 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:08:47.920320 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:08:47.921386 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:08:47.921391 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:08:47.922849 (+ 1458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:08:47.922860 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:08:47.923927 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:08:47.923932 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:08:47.924100 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:08:47.924122 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 15:08:47.924221 (+ 99us) client_negotiation.cc:240] Negotiation successful
0926 15:08:47.924344 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":293,"threads_started":1}
I20260926 15:08:47.925485 13279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de3763cd55b04f0088c4b8f68d157ff2" candidate_uuid: "0f54da405e424f52a4f1e4dd91d06fd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "872a2f2a4bf1419b89b7dbdd81ff8745" is_pre_election: true
I20260926 15:08:47.925680 13412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de3763cd55b04f0088c4b8f68d157ff2" candidate_uuid: "0f54da405e424f52a4f1e4dd91d06fd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed1a343d43004d34b76cdab47959592f" is_pre_election: true
I20260926 15:08:47.926049 13279 raft_consensus.cc:2454] T de3763cd55b04f0088c4b8f68d157ff2 P 872a2f2a4bf1419b89b7dbdd81ff8745 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f54da405e424f52a4f1e4dd91d06fd0 in term 0.
I20260926 15:08:47.926141 13412 raft_consensus.cc:2454] T de3763cd55b04f0088c4b8f68d157ff2 P ed1a343d43004d34b76cdab47959592f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f54da405e424f52a4f1e4dd91d06fd0 in term 0.
I20260926 15:08:47.926887 13080 leader_election.cc:302] T de3763cd55b04f0088c4b8f68d157ff2 P 0f54da405e424f52a4f1e4dd91d06fd0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f54da405e424f52a4f1e4dd91d06fd0, 872a2f2a4bf1419b89b7dbdd81ff8745; no voters:
I20260926 15:08:47.927245 13482 raft_consensus.cc:2781] T de3763cd55b04f0088c4b8f68d157ff2 P 0f54da405e424f52a4f1e4dd91d06fd0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 15:08:47.927397 13482 raft_consensus.cc:482] T de3763cd55b04f0088c4b8f68d157ff2 P 0f54da405e424f52a4f1e4dd91d06fd0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 15:08:47.927498 13482 raft_consensus.cc:3037] T de3763cd55b04f0088c4b8f68d157ff2 P 0f54da405e424f52a4f1e4dd91d06fd0 [term 0 FOLLOWER]: Advancing to term 1
W20260926 15:08:47.927851 13191 tablet.cc:2369] T de3763cd55b04f0088c4b8f68d157ff2 P 0f54da405e424f52a4f1e4dd91d06fd0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 15:08:47.929900 13482 raft_consensus.cc:504] T de3763cd55b04f0088c4b8f68d157ff2 P 0f54da405e424f52a4f1e4dd91d06fd0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f54da405e424f52a4f1e4dd91d06fd0" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 44113 } } peers { permanent_uuid: "872a2f2a4bf1419b89b7dbdd81ff8745" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 36713 } } peers { permanent_uuid: "ed1a343d43004d34b76cdab47959592f" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 35591 } }
I20260926 15:08:47.930564 13482 leader_election.cc:288] T de3763cd55b04f0088c4b8f68d157ff2 P 0f54da405e424f52a4f1e4dd91d06fd0 [CANDIDATE]: Term 1 election: Requested vote from peers 872a2f2a4bf1419b89b7dbdd81ff8745 (127.12.34.194:36713), ed1a343d43004d34b76cdab47959592f (127.12.34.195:35591)
I20260926 15:08:47.931130 13279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de3763cd55b04f0088c4b8f68d157ff2" candidate_uuid: "0f54da405e424f52a4f1e4dd91d06fd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "872a2f2a4bf1419b89b7dbdd81ff8745"
I20260926 15:08:47.931130 13412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de3763cd55b04f0088c4b8f68d157ff2" candidate_uuid: "0f54da405e424f52a4f1e4dd91d06fd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed1a343d43004d34b76cdab47959592f"
I20260926 15:08:47.931339 13279 raft_consensus.cc:3037] T de3763cd55b04f0088c4b8f68d157ff2 P 872a2f2a4bf1419b89b7dbdd81ff8745 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:08:47.931339 13412 raft_consensus.cc:3037] T de3763cd55b04f0088c4b8f68d157ff2 P ed1a343d43004d34b76cdab47959592f [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:08:47.934530 13279 raft_consensus.cc:2454] T de3763cd55b04f0088c4b8f68d157ff2 P 872a2f2a4bf1419b89b7dbdd81ff8745 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f54da405e424f52a4f1e4dd91d06fd0 in term 1.
I20260926 15:08:47.934547 13412 raft_consensus.cc:2454] T de3763cd55b04f0088c4b8f68d157ff2 P ed1a343d43004d34b76cdab47959592f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f54da405e424f52a4f1e4dd91d06fd0 in term 1.
I20260926 15:08:47.935048 13080 leader_election.cc:302] T de3763cd55b04f0088c4b8f68d157ff2 P 0f54da405e424f52a4f1e4dd91d06fd0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f54da405e424f52a4f1e4dd91d06fd0, 872a2f2a4bf1419b89b7dbdd81ff8745; no voters:
I20260926 15:08:47.935380 13482 raft_consensus.cc:2781] T de3763cd55b04f0088c4b8f68d157ff2 P 0f54da405e424f52a4f1e4dd91d06fd0 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:08:47.935927 13482 raft_consensus.cc:686] T de3763cd55b04f0088c4b8f68d157ff2 P 0f54da405e424f52a4f1e4dd91d06fd0 [term 1 LEADER]: Becoming Leader. State: Replica: 0f54da405e424f52a4f1e4dd91d06fd0, State: Running, Role: LEADER
I20260926 15:08:47.936403 13482 consensus_queue.cc:237] T de3763cd55b04f0088c4b8f68d157ff2 P 0f54da405e424f52a4f1e4dd91d06fd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f54da405e424f52a4f1e4dd91d06fd0" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 44113 } } peers { permanent_uuid: "872a2f2a4bf1419b89b7dbdd81ff8745" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 36713 } } peers { permanent_uuid: "ed1a343d43004d34b76cdab47959592f" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 35591 } }
I20260926 15:08:47.941165 13001 catalog_manager.cc:5495] T de3763cd55b04f0088c4b8f68d157ff2 P 0f54da405e424f52a4f1e4dd91d06fd0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0f54da405e424f52a4f1e4dd91d06fd0 (127.12.34.193). New cstate: current_term: 1 leader_uuid: "0f54da405e424f52a4f1e4dd91d06fd0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f54da405e424f52a4f1e4dd91d06fd0" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 44113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "872a2f2a4bf1419b89b7dbdd81ff8745" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 36713 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ed1a343d43004d34b76cdab47959592f" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 35591 } health_report { overall_health: UNKNOWN } } }
Sep 26 15:08:47 dist-test-slave-865v krb5kdc[12956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435325, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.34.193@KRBTEST.COM
I20260926 15:08:48.003748 13470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:08:47.986188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43504 (local address 127.12.34.193:44113)
0926 15:08:47.986483 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0926 15:08:47.986491 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 15:08:47.986832 (+ 341us) server_negotiation.cc:407] Connection header received
0926 15:08:47.987077 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:08:47.987082 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:08:47.987148 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:08:47.987277 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:08:47.989031 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:08:47.989865 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:08:47.991179 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:08:47.991654 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:08:47.991797 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:08:47.996429 (+ 4632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:08:47.996444 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:08:47.996461 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:08:47.996494 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:08:48.000252 (+ 3758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:08:48.001521 (+ 1269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:08:48.001532 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:08:48.001536 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:08:48.001755 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:08:48.002180 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:08:48.002186 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:08:48.002189 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:08:48.002558 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:08:48.002878 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0926 15:08:48.003149 (+ 271us) server_negotiation.cc:299] Negotiation successful
0926 15:08:48.003426 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111}
Sep 26 15:08:48 dist-test-slave-865v krb5kdc[12956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435328, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 15:08:48 dist-test-slave-865v krb5kdc[12956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435328, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.12.34.193@KRBTEST.COM
I20260926 15:08:48.050427 13470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:08:48.035905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43506 (local address 127.12.34.193:44113)
0926 15:08:48.036165 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0926 15:08:48.036173 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 15:08:48.036427 (+ 254us) server_negotiation.cc:407] Connection header received
0926 15:08:48.036678 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:08:48.036684 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:08:48.036756 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:08:48.036863 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:08:48.038195 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:08:48.038912 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:08:48.040452 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:08:48.040878 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:08:48.041024 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:08:48.045796 (+ 4772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:08:48.045801 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:08:48.045804 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:08:48.045835 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:08:48.048208 (+ 2373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:08:48.048770 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:08:48.048774 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:08:48.048776 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:08:48.048887 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:08:48.049285 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:08:48.049288 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:08:48.049290 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:08:48.049552 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:08:48.049750 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0926 15:08:48.049927 (+ 177us) server_negotiation.cc:299] Negotiation successful
0926 15:08:48.050079 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
Sep 26 15:08:48 dist-test-slave-865v krb5kdc[12956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435328, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 15:08:48 dist-test-slave-865v krb5kdc[12956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435328, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.34.193@KRBTEST.COM
I20260926 15:08:48.094591 13470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:08:48.080974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43512 (local address 127.12.34.193:44113)
0926 15:08:48.081234 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0926 15:08:48.081239 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:08:48.081321 (+ 82us) server_negotiation.cc:407] Connection header received
0926 15:08:48.081489 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:08:48.081494 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:08:48.081573 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:08:48.081690 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:08:48.083119 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:08:48.083884 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:08:48.085161 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:08:48.085734 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:08:48.085869 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:08:48.089580 (+ 3711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:08:48.089585 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:08:48.089588 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:08:48.089616 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:08:48.092117 (+ 2501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:08:48.092745 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:08:48.092749 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:08:48.092752 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:08:48.092864 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:08:48.093354 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:08:48.093361 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:08:48.093368 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:08:48.093710 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:08:48.093904 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0926 15:08:48.094186 (+ 282us) server_negotiation.cc:299] Negotiation successful
0926 15:08:48.094340 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
W20260926 15:08:48.096338 13105 tablet_service.cc:2908] Rejecting scan request for tablet de3763cd55b04f0088c4b8f68d157ff2: Uninitialized: clean time has not yet been initialized
I20260926 15:08:48.099097 12427 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 13062
I20260926 15:08:48.113621 13470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:08:48.100085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43522 (local address 127.12.34.193:44113)
0926 15:08:48.100270 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0926 15:08:48.100276 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:08:48.100537 (+ 261us) server_negotiation.cc:407] Connection header received
0926 15:08:48.100748 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:08:48.100754 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:08:48.100833 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:08:48.100916 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:08:48.102410 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:08:48.103095 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:08:48.104778 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:08:48.105214 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:08:48.105303 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:08:48.108411 (+ 3108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:08:48.108415 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:08:48.108418 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:08:48.108444 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:08:48.110869 (+ 2425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:08:48.111584 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:08:48.111588 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:08:48.111590 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:08:48.111712 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:08:48.112373 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:08:48.112376 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:08:48.112378 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:08:48.112613 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:08:48.112786 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0926 15:08:48.113153 (+ 367us) server_negotiation.cc:299] Negotiation successful
0926 15:08:48.113329 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 15:08:48.114313 13185 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:08:48.205677 12427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8uoJGv/build/asan/bin/kudu with pid 13062
I20260926 15:08:48.222872 12427 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 13196
Sep 26 15:08:48 dist-test-slave-865v krb5kdc[12956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435328, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.34.194@KRBTEST.COM
I20260926 15:08:48.239177 13472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:08:48.224019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60438 (local address 127.12.34.194:36713)
0926 15:08:48.224207 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0926 15:08:48.224211 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:08:48.224230 (+ 19us) server_negotiation.cc:407] Connection header received
0926 15:08:48.224284 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:08:48.224287 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:08:48.224344 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:08:48.224433 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:08:48.225694 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:08:48.226456 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:08:48.227740 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:08:48.228146 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:08:48.228246 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:08:48.232095 (+ 3849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:08:48.232107 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:08:48.232123 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:08:48.232158 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:08:48.235159 (+ 3001us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:08:48.235806 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:08:48.235813 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:08:48.235815 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:08:48.236382 (+ 567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:08:48.237218 (+ 836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:08:48.237229 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:08:48.237235 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:08:48.237884 (+ 649us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:08:48.238236 (+ 352us) server_negotiation.cc:1035] Waiting for connection context
0926 15:08:48.238561 (+ 325us) server_negotiation.cc:299] Negotiation successful
0926 15:08:48.238902 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260926 15:08:48.239951 13319 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:08:48.327833 12427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8uoJGv/build/asan/bin/kudu with pid 13196
I20260926 15:08:48.344319 12427 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 13329
Sep 26 15:08:48 dist-test-slave-865v krb5kdc[12956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435328, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.34.195@KRBTEST.COM
I20260926 15:08:48.358361 13474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:08:48.345064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35536 (local address 127.12.34.195:35591)
0926 15:08:48.345289 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0926 15:08:48.345294 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:08:48.345311 (+ 17us) server_negotiation.cc:407] Connection header received
0926 15:08:48.345407 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:08:48.345411 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:08:48.345470 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:08:48.345547 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:08:48.346729 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:08:48.347440 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:08:48.348734 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:08:48.349269 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:08:48.349368 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:08:48.353299 (+ 3931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:08:48.353313 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:08:48.353329 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:08:48.353362 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:08:48.355922 (+ 2560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:08:48.356548 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:08:48.356555 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:08:48.356557 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:08:48.356683 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:08:48.356991 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:08:48.356994 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:08:48.356996 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:08:48.357383 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:08:48.357600 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0926 15:08:48.357848 (+ 248us) server_negotiation.cc:299] Negotiation successful
0926 15:08:48.357993 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118}
I20260926 15:08:48.359001 13452 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:08:48.444927 12427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8uoJGv/build/asan/bin/kudu with pid 13329
I20260926 15:08:48.461055 12427 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 12969
I20260926 15:08:48.462219 13031 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:08:48.567039 12427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8uoJGv/build/asan/bin/kudu with pid 12969
2026-09-26T15:08:48Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4128 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435318278681-12427-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 15:08:48 dist-test-slave-865v krb5kdc[13505](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 15:08:48 dist-test-slave-865v krb5kdc[13505](info): set up 2 sockets
Sep 26 15:08:48 dist-test-slave-865v krb5kdc[13505](info): commencing operation
krb5kdc: starting...
W20260926 15:08:49.657145 12427 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.000s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435318278681-12427-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435318278681-12427-0/krb5kdc/test-user.keytab.
Sep 26 15:08:49 dist-test-slave-865v krb5kdc[13505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435329, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T15:08:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T15:08:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.34.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.34.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.34.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.254.keytab.
Entry for principal kudu/127.12.34.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.254.keytab.
I20260926 15:08:49.840953 12427 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8uoJGv/build/asan/bin/kudu
/tmp/dist-test-task8uoJGv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435318278681-12427-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435318278681-12427-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435318278681-12427-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.34.254:40761
--webserver_interface=127.12.34.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.254.keytab
--principal=kudu/127.12.34.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.34.212:46705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.34.254:40761
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435318278681-12427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435318278681-12427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435318278681-12427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:08:50.161945 13518 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:08:50.162246 13518 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:08:50.162420 13518 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:08:50.169989 13518 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:08:50.170095 13518 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 15:08:50.170135 13518 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:08:50.170230 13518 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:08:50.170306 13518 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:08:50.170528 13518 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:08:50.180380 13518 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.34.212:46705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435318278681-12427-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435318278681-12427-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.34.254:40761
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.254.keytab
--principal=kudu/127.12.34.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.34.254:40761
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.34.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435318278681-12427-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:08:50.181697 13518 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:08:50.182981 13518 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:08:50.196748 13524 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:08:50.196815 13523 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:08:50.196847 13526 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:08:50.197943 13518 server_base.cc:1034] running on GCE node
I20260926 15:08:50.198693 13518 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:08:50.200119 13518 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:08:50.201360 13518 hybrid_clock.cc:648] HybridClock initialized: now 1790435330201294 us; error 44 us; skew 500 ppm
I20260926 15:08:50.201706 13518 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:08:50 dist-test-slave-865v krb5kdc[13505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435330, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:08:50.204962 13518 init.cc:377] Logged in from keytab as kudu/127.12.34.254@KRBTEST.COM (short username kudu)
I20260926 15:08:50.206766 13518 webserver.cc:460] Webserver started at http://127.12.34.254:35013/ using document root <none> and password file <none>
I20260926 15:08:50.207499 13518 fs_manager.cc:359] Metadata directory not provided
I20260926 15:08:50.207630 13518 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:08:50.207949 13518 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:08:50.210896 13518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435318278681-12427-0/minicluster-data/master-0/data/instance:
uuid: "e855e2d27ca44f7fbc65d11d9bc0b942"
format_stamp: "Formatted at 2026-09-26 15:08:50 on dist-test-slave-865v"
server_key: "8e6489158e1bd11804f98d7b4de4f90e"
server_key_iv: "310a3aef65e11082dbdeb403c723a04e"
server_key_version: "encryptionkey@0"
I20260926 15:08:50.211673 13518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435318278681-12427-0/minicluster-data/master-0/wal/instance:
uuid: "e855e2d27ca44f7fbc65d11d9bc0b942"
format_stamp: "Formatted at 2026-09-26 15:08:50 on dist-test-slave-865v"
server_key: "8e6489158e1bd11804f98d7b4de4f90e"
server_key_iv: "310a3aef65e11082dbdeb403c723a04e"
server_key_version: "encryptionkey@0"
I20260926 15:08:50.217618 13518 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.002s
I20260926 15:08:50.222155 13533 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:08:50.223874 13518 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260926 15:08:50.224079 13518 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435318278681-12427-0/minicluster-data/master-0/data,/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435318278681-12427-0/minicluster-data/master-0/wal
uuid: "e855e2d27ca44f7fbc65d11d9bc0b942"
format_stamp: "Formatted at 2026-09-26 15:08:50 on dist-test-slave-865v"
server_key: "8e6489158e1bd11804f98d7b4de4f90e"
server_key_iv: "310a3aef65e11082dbdeb403c723a04e"
server_key_version: "encryptionkey@0"
I20260926 15:08:50.224306 13518 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435318278681-12427-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435318278681-12427-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435318278681-12427-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:08:50.235827 13518 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:08:50.240115 13518 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:08:50.240381 13518 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:08:50.263722 13518 rpc_server.cc:307] RPC server started. Bound to: 127.12.34.254:40761
I20260926 15:08:50.263906 13594 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.34.254:40761 every 8 connection(s)
I20260926 15:08:50.265393 13518 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
I20260926 15:08:50.269098 13595 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:08:50.269346 12427 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8uoJGv/build/asan/bin/kudu as pid 13518
I20260926 15:08:50.269577 12427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435318278681-12427-0/minicluster-data/master-0/wal/instance
I20260926 15:08:50.270213 12427 external_mini_cluster.cc:1428] Setting key a44ea33fa431fb322ed3a75167ced324
I20260926 15:08:50.283288 13595 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e855e2d27ca44f7fbc65d11d9bc0b942: Bootstrap starting.
I20260926 15:08:50.286405 13595 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e855e2d27ca44f7fbc65d11d9bc0b942: Neither blocks nor log segments found. Creating new log.
Sep 26 15:08:50 dist-test-slave-865v krb5kdc[13505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435329, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.34.254@KRBTEST.COM
I20260926 15:08:50.287611 13595 log.cc:824] T 00000000000000000000000000000000 P e855e2d27ca44f7fbc65d11d9bc0b942: Log is configured to *not* fsync() on all Append() calls
I20260926 15:08:50.291272 13595 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e855e2d27ca44f7fbc65d11d9bc0b942: No bootstrap required, opened a new log
I20260926 15:08:50.297667 13598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:08:50.272765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57062 (local address 127.12.34.254:40761)
0926 15:08:50.273609 (+ 844us) server_negotiation.cc:206] Beginning negotiation
0926 15:08:50.273625 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 15:08:50.273684 (+ 59us) server_negotiation.cc:407] Connection header received
0926 15:08:50.275811 (+ 2127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:08:50.275839 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:08:50.276364 (+ 525us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:08:50.276796 (+ 432us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:08:50.278588 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:08:50.280658 (+ 2070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:08:50.282082 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:08:50.283083 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:08:50.283278 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:08:50.289217 (+ 5939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:08:50.289229 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:08:50.289248 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:08:50.289297 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:08:50.292948 (+ 3651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:08:50.293916 (+ 968us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:08:50.293923 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:08:50.293932 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:08:50.294126 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:08:50.294688 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:08:50.294692 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:08:50.294694 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:08:50.295243 (+ 549us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:08:50.295459 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0926 15:08:50.296036 (+ 577us) server_negotiation.cc:299] Negotiation successful
0926 15:08:50.296560 (+ 524us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":619,"thread_start_us":385,"threads_started":1}
I20260926 15:08:50.302547 13595 raft_consensus.cc:348] T 00000000000000000000000000000000 P e855e2d27ca44f7fbc65d11d9bc0b942 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e855e2d27ca44f7fbc65d11d9bc0b942" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 40761 } }
I20260926 15:08:50.303068 13595 raft_consensus.cc:374] T 00000000000000000000000000000000 P e855e2d27ca44f7fbc65d11d9bc0b942 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:08:50.303375 13595 raft_consensus.cc:729] T 00000000000000000000000000000000 P e855e2d27ca44f7fbc65d11d9bc0b942 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e855e2d27ca44f7fbc65d11d9bc0b942, State: Initialized, Role: FOLLOWER
I20260926 15:08:50.304152 13595 consensus_queue.cc:260] T 00000000000000000000000000000000 P e855e2d27ca44f7fbc65d11d9bc0b942 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e855e2d27ca44f7fbc65d11d9bc0b942" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 40761 } }
I20260926 15:08:50.304450 13595 raft_consensus.cc:388] T 00000000000000000000000000000000 P e855e2d27ca44f7fbc65d11d9bc0b942 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 15:08:50.304598 13595 raft_consensus.cc:482] T 00000000000000000000000000000000 P e855e2d27ca44f7fbc65d11d9bc0b942 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 15:08:50.304782 13595 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e855e2d27ca44f7fbc65d11d9bc0b942 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:08:50.307284 13595 raft_consensus.cc:504] T 00000000000000000000000000000000 P e855e2d27ca44f7fbc65d11d9bc0b942 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e855e2d27ca44f7fbc65d11d9bc0b942" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 40761 } }
I20260926 15:08:50.307978 13595 leader_election.cc:302] T 00000000000000000000000000000000 P e855e2d27ca44f7fbc65d11d9bc0b942 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e855e2d27ca44f7fbc65d11d9bc0b942; no voters:
I20260926 15:08:50.308645 13595 leader_election.cc:288] T 00000000000000000000000000000000 P e855e2d27ca44f7fbc65d11d9bc0b942 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 15:08:50.308923 13600 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e855e2d27ca44f7fbc65d11d9bc0b942 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:08:50.309629 13600 raft_consensus.cc:686] T 00000000000000000000000000000000 P e855e2d27ca44f7fbc65d11d9bc0b942 [term 1 LEADER]: Becoming Leader. State: Replica: e855e2d27ca44f7fbc65d11d9bc0b942, State: Running, Role: LEADER
I20260926 15:08:50.310153 13600 consensus_queue.cc:237] T 00000000000000000000000000000000 P e855e2d27ca44f7fbc65d11d9bc0b942 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e855e2d27ca44f7fbc65d11d9bc0b942" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 40761 } }
I20260926 15:08:50.310871 13595 sys_catalog.cc:564] T 00000000000000000000000000000000 P e855e2d27ca44f7fbc65d11d9bc0b942 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:08:50.313273 13602 sys_catalog.cc:455] T 00000000000000000000000000000000 P e855e2d27ca44f7fbc65d11d9bc0b942 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e855e2d27ca44f7fbc65d11d9bc0b942. Latest consensus state: current_term: 1 leader_uuid: "e855e2d27ca44f7fbc65d11d9bc0b942" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e855e2d27ca44f7fbc65d11d9bc0b942" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 40761 } } }
I20260926 15:08:50.313172 13601 sys_catalog.cc:455] T 00000000000000000000000000000000 P e855e2d27ca44f7fbc65d11d9bc0b942 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e855e2d27ca44f7fbc65d11d9bc0b942" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e855e2d27ca44f7fbc65d11d9bc0b942" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 40761 } } }
I20260926 15:08:50.313748 13601 sys_catalog.cc:458] T 00000000000000000000000000000000 P e855e2d27ca44f7fbc65d11d9bc0b942 [sys.catalog]: This master's current role is: LEADER
I20260926 15:08:50.313748 13602 sys_catalog.cc:458] T 00000000000000000000000000000000 P e855e2d27ca44f7fbc65d11d9bc0b942 [sys.catalog]: This master's current role is: LEADER
I20260926 15:08:50.314909 13606 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:08:50.321463 13606 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 15:08:50.331321 13606 catalog_manager.cc:1347] Generated new cluster ID: 5156901604924395b8d91917539a3eb5
I20260926 15:08:50.331441 13606 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 15:08:50.341653 13606 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 15:08:50.343528 13606 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:08:50.351747 13606 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e855e2d27ca44f7fbc65d11d9bc0b942: Generated new TSK 0
I20260926 15:08:50.352749 13606 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.34.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.34.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.34.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.193.keytab.
Entry for principal kudu/127.12.34.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.193.keytab.
I20260926 15:08:50.405537 12427 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8uoJGv/build/asan/bin/kudu
/tmp/dist-test-task8uoJGv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435318278681-12427-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435318278681-12427-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435318278681-12427-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435318278681-12427-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.34.193:0
--local_ip_for_outbound_sockets=127.12.34.193
--webserver_interface=127.12.34.193
--webserver_port=0
--tserver_master_addrs=127.12.34.254:40761
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.193.keytab
--principal=kudu/127.12.34.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.34.212:46705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435318278681-12427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435318278681-12427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435318278681-12427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:08:50.746330 13621 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:08:50.746608 13621 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:08:50.746706 13621 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:08:50.753865 13621 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:08:50.753974 13621 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:08:50.754061 13621 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 15:08:50.754143 13621 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.34.193
I20260926 15:08:50.763538 13621 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.34.212:46705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435318278681-12427-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435318278681-12427-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.193.keytab
--principal=kudu/127.12.34.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.34.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435318278681-12427-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.34.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.12.34.254:40761
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.34.193
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435318278681-12427-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:08:50.764694 13621 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:08:50.765990 13621 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:08:50.779011 13626 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:08:50.779177 13627 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:08:50.779153 13629 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:08:50.779758 13621 server_base.cc:1034] running on GCE node
I20260926 15:08:50.780325 13621 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:08:50.781270 13621 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:08:50.782521 13621 hybrid_clock.cc:648] HybridClock initialized: now 1790435330782456 us; error 41 us; skew 500 ppm
I20260926 15:08:50.782826 13621 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:08:50 dist-test-slave-865v krb5kdc[13505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435330, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:08:50.785929 13621 init.cc:377] Logged in from keytab as kudu/127.12.34.193@KRBTEST.COM (short username kudu)
I20260926 15:08:50.787489 13621 webserver.cc:460] Webserver started at http://127.12.34.193:41567/ using document root <none> and password file <none>
I20260926 15:08:50.788184 13621 fs_manager.cc:359] Metadata directory not provided
I20260926 15:08:50.788311 13621 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:08:50.788630 13621 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:08:50.791507 13621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435318278681-12427-0/minicluster-data/ts-0/data/instance:
uuid: "a74c4495c62e440bbf55cd02a879911c"
format_stamp: "Formatted at 2026-09-26 15:08:50 on dist-test-slave-865v"
server_key: "4d861754cb482e7020a3bc9f1933ce97"
server_key_iv: "2c2b184e33b7c4c3f58dcf1a0165fb0c"
server_key_version: "encryptionkey@0"
I20260926 15:08:50.792305 13621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435318278681-12427-0/minicluster-data/ts-0/wal/instance:
uuid: "a74c4495c62e440bbf55cd02a879911c"
format_stamp: "Formatted at 2026-09-26 15:08:50 on dist-test-slave-865v"
server_key: "4d861754cb482e7020a3bc9f1933ce97"
server_key_iv: "2c2b184e33b7c4c3f58dcf1a0165fb0c"
server_key_version: "encryptionkey@0"
I20260926 15:08:50.797536 13621 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260926 15:08:50.801584 13636 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:08:50.802939 13621 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 15:08:50.803115 13621 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435318278681-12427-0/minicluster-data/ts-0/data,/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435318278681-12427-0/minicluster-data/ts-0/wal
uuid: "a74c4495c62e440bbf55cd02a879911c"
format_stamp: "Formatted at 2026-09-26 15:08:50 on dist-test-slave-865v"
server_key: "4d861754cb482e7020a3bc9f1933ce97"
server_key_iv: "2c2b184e33b7c4c3f58dcf1a0165fb0c"
server_key_version: "encryptionkey@0"
I20260926 15:08:50.803311 13621 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435318278681-12427-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435318278681-12427-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435318278681-12427-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:08:50.818786 13621 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:08:50.822695 13621 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:08:50.822957 13621 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:08:50.825469 13621 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:08:50.825582 13621 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:08:50.825744 13621 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:08:50.825824 13621 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 15:08:50 dist-test-slave-865v krb5kdc[13505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435330, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.193@KRBTEST.COM for kudu/127.12.34.254@KRBTEST.COM
I20260926 15:08:50.849781 13645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:08:50.826199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.193:44663 (local address 127.12.34.254:40761)
0926 15:08:50.826727 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0926 15:08:50.826733 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:08:50.827951 (+ 1218us) server_negotiation.cc:407] Connection header received
0926 15:08:50.830305 (+ 2354us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:08:50.830315 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:08:50.830421 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:08:50.830585 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:08:50.832920 (+ 2335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:08:50.833966 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:08:50.836334 (+ 2368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:08:50.836795 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:08:50.836938 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:08:50.842364 (+ 5426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:08:50.842369 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:08:50.842373 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:08:50.842405 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:08:50.845807 (+ 3402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:08:50.846851 (+ 1044us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:08:50.846858 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:08:50.846862 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:08:50.847020 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:08:50.847623 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:08:50.847632 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:08:50.847638 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:08:50.848098 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:08:50.848303 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0926 15:08:50.849211 (+ 908us) server_negotiation.cc:299] Negotiation successful
0926 15:08:50.849418 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":416,"thread_start_us":319,"threads_started":1}
I20260926 15:08:50.850876 13646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:08:50.826878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:40761 (local address 127.12.34.193:44663)
0926 15:08:50.827810 (+ 932us) negotiation.cc:107] Waiting for socket to connect
0926 15:08:50.827859 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 15:08:50.829943 (+ 2084us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:08:50.830840 (+ 897us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:08:50.830854 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:08:50.831777 (+ 923us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:08:50.832757 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:08:50.832776 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:08:50.834118 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:08:50.834134 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:08:50.836196 (+ 2062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:08:50.836208 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:08:50.837144 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:08:50.837155 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:08:50.837516 (+ 361us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:08:50.838449 (+ 933us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:08:50.838488 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:08:50.842076 (+ 3588us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:08:50.846018 (+ 3942us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:08:50.846027 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:08:50.846049 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:08:50.846649 (+ 600us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:08:50.847133 (+ 484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:08:50.847138 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:08:50.847142 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:08:50.847474 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:08:50.848255 (+ 781us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:08:50.848261 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:08:50.848734 (+ 473us) client_negotiation.cc:769] Sending connection context
0926 15:08:50.849138 (+ 404us) client_negotiation.cc:240] Negotiation successful
0926 15:08:50.849683 (+ 545us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":643,"thread_start_us":437,"threads_started":1}
I20260926 15:08:50.874784 13756 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.34.193:39853 every 8 connection(s)
I20260926 15:08:50.874814 13621 rpc_server.cc:307] RPC server started. Bound to: 127.12.34.193:39853
I20260926 15:08:50.876181 13621 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435318278681-12427-0/minicluster-data/ts-0/data/info.pb
I20260926 15:08:50.878288 13757 heartbeater.cc:347] Connected to a master server at 127.12.34.254:40761
I20260926 15:08:50.878537 13757 heartbeater.cc:464] Registering TS with master...
I20260926 15:08:50.879211 13757 heartbeater.cc:511] Master 127.12.34.254:40761 requested a full tablet report, sending...
I20260926 15:08:50.881585 13550 ts_manager.cc:194] Registered new tserver with Master: a74c4495c62e440bbf55cd02a879911c (127.12.34.193:39853)
I20260926 15:08:50.883399 13550 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.34.193@KRBTEST.COM'} at 127.12.34.193:44663
I20260926 15:08:50.883728 12427 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8uoJGv/build/asan/bin/kudu as pid 13621
I20260926 15:08:50.883958 12427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435318278681-12427-0/minicluster-data/ts-0/wal/instance
I20260926 15:08:50.884719 12427 external_mini_cluster.cc:1428] Setting key 67ac3d7ee162045a0a8996b53319e4bd
I20260926 15:08:50.885773 13645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:08:50.865159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.193:46629 (local address 127.12.34.254:40761)
0926 15:08:50.865405 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0926 15:08:50.865414 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 15:08:50.872121 (+ 6707us) server_negotiation.cc:407] Connection header received
0926 15:08:50.872287 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:08:50.872296 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:08:50.872392 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:08:50.872549 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:08:50.873911 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:08:50.875093 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:08:50.876354 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:08:50.876937 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:08:50.877122 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:08:50.879354 (+ 2232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:08:50.879367 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:08:50.879371 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:08:50.879407 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:08:50.882801 (+ 3394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:08:50.883612 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:08:50.883618 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:08:50.883623 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:08:50.883798 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:08:50.884294 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:08:50.884299 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:08:50.884302 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:08:50.884623 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:08:50.884769 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0926 15:08:50.885212 (+ 443us) server_negotiation.cc:299] Negotiation successful
0926 15:08:50.885425 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 15:08:50.886123 13742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:08:50.864975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:40761 (local address 127.12.34.193:46629)
0926 15:08:50.871984 (+ 7009us) negotiation.cc:107] Waiting for socket to connect
0926 15:08:50.872005 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 15:08:50.872136 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:08:50.872719 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:08:50.872725 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:08:50.873287 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:08:50.873720 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:08:50.873734 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:08:50.875221 (+ 1487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:08:50.875224 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:08:50.876207 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:08:50.876219 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:08:50.877261 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:08:50.877271 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:08:50.877571 (+ 300us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:08:50.878323 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:08:50.878340 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:08:50.879171 (+ 831us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:08:50.882942 (+ 3771us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:08:50.882946 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:08:50.882949 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:08:50.883444 (+ 495us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:08:50.883956 (+ 512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:08:50.883961 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:08:50.883964 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:08:50.884125 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:08:50.884750 (+ 625us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:08:50.884755 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:08:50.884930 (+ 175us) client_negotiation.cc:769] Sending connection context
0926 15:08:50.885073 (+ 143us) client_negotiation.cc:240] Negotiation successful
0926 15:08:50.885673 (+ 600us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6840,"thread_start_us":6535,"threads_started":1}
WARNING: no policy specified for kudu/127.12.34.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.34.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.34.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.194.keytab.
Entry for principal kudu/127.12.34.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.194.keytab.
I20260926 15:08:50.926982 12427 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8uoJGv/build/asan/bin/kudu
/tmp/dist-test-task8uoJGv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435318278681-12427-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435318278681-12427-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435318278681-12427-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435318278681-12427-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.34.194:0
--local_ip_for_outbound_sockets=127.12.34.194
--webserver_interface=127.12.34.194
--webserver_port=0
--tserver_master_addrs=127.12.34.254:40761
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.194.keytab
--principal=kudu/127.12.34.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.34.212:46705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435318278681-12427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435318278681-12427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435318278681-12427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:08:51.241614 13761 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:08:51.241938 13761 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:08:51.242110 13761 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:08:51.249214 13761 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:08:51.249364 13761 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:08:51.249518 13761 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 15:08:51.249653 13761 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.34.194
I20260926 15:08:51.259171 13761 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.34.212:46705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435318278681-12427-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435318278681-12427-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.194.keytab
--principal=kudu/127.12.34.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.34.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435318278681-12427-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.34.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.12.34.254:40761
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.34.194
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435318278681-12427-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:08:51.260435 13761 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:08:51.261713 13761 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:08:51.274296 13767 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:08:51.274528 13766 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:08:51.275064 13761 server_base.cc:1034] running on GCE node
W20260926 15:08:51.274422 13769 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:08:51.275817 13761 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:08:51.276830 13761 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:08:51.278069 13761 hybrid_clock.cc:648] HybridClock initialized: now 1790435331277974 us; error 60 us; skew 500 ppm
I20260926 15:08:51.278375 13761 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:08:51 dist-test-slave-865v krb5kdc[13505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435331, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:08:51.289717 13761 init.cc:377] Logged in from keytab as kudu/127.12.34.194@KRBTEST.COM (short username kudu)
I20260926 15:08:51.291168 13761 webserver.cc:460] Webserver started at http://127.12.34.194:44511/ using document root <none> and password file <none>
I20260926 15:08:51.291875 13761 fs_manager.cc:359] Metadata directory not provided
I20260926 15:08:51.291958 13761 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:08:51.292205 13761 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:08:51.294981 13761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435318278681-12427-0/minicluster-data/ts-1/data/instance:
uuid: "0be73b1f9d444997ab7d9ff8ef11a74f"
format_stamp: "Formatted at 2026-09-26 15:08:51 on dist-test-slave-865v"
server_key: "f872dc789a8edb290f57cbd980159019"
server_key_iv: "884b9556077f918a62c2833ce0a37ee6"
server_key_version: "encryptionkey@0"
I20260926 15:08:51.295675 13761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435318278681-12427-0/minicluster-data/ts-1/wal/instance:
uuid: "0be73b1f9d444997ab7d9ff8ef11a74f"
format_stamp: "Formatted at 2026-09-26 15:08:51 on dist-test-slave-865v"
server_key: "f872dc789a8edb290f57cbd980159019"
server_key_iv: "884b9556077f918a62c2833ce0a37ee6"
server_key_version: "encryptionkey@0"
I20260926 15:08:51.300932 13761 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 15:08:51.304493 13776 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:08:51.305984 13761 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 15:08:51.306133 13761 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435318278681-12427-0/minicluster-data/ts-1/data,/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435318278681-12427-0/minicluster-data/ts-1/wal
uuid: "0be73b1f9d444997ab7d9ff8ef11a74f"
format_stamp: "Formatted at 2026-09-26 15:08:51 on dist-test-slave-865v"
server_key: "f872dc789a8edb290f57cbd980159019"
server_key_iv: "884b9556077f918a62c2833ce0a37ee6"
server_key_version: "encryptionkey@0"
I20260926 15:08:51.306354 13761 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435318278681-12427-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435318278681-12427-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435318278681-12427-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:08:51.323272 13761 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:08:51.327371 13761 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:08:51.327658 13761 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:08:51.330581 13761 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:08:51.330663 13761 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:08:51.330792 13761 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:08:51.330855 13761 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 15:08:51 dist-test-slave-865v krb5kdc[13505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435331, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.194@KRBTEST.COM for kudu/127.12.34.254@KRBTEST.COM
I20260926 15:08:51.354660 13645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:08:51.331269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.194:57103 (local address 127.12.34.254:40761)
0926 15:08:51.331417 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0926 15:08:51.331421 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:08:51.332907 (+ 1486us) server_negotiation.cc:407] Connection header received
0926 15:08:51.335195 (+ 2288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:08:51.335201 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:08:51.335284 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:08:51.335366 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:08:51.337875 (+ 2509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:08:51.338904 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:08:51.341237 (+ 2333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:08:51.342030 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:08:51.342184 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:08:51.346456 (+ 4272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:08:51.346463 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:08:51.346467 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:08:51.346509 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:08:51.350598 (+ 4089us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:08:51.351602 (+ 1004us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:08:51.351609 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:08:51.351614 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:08:51.351836 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:08:51.352510 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:08:51.352515 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:08:51.352519 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:08:51.352874 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:08:51.353072 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0926 15:08:51.354163 (+ 1091us) server_negotiation.cc:299] Negotiation successful
0926 15:08:51.354420 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 15:08:51.356196 13785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:08:51.331838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:40761 (local address 127.12.34.194:57103)
0926 15:08:51.332730 (+ 892us) negotiation.cc:107] Waiting for socket to connect
0926 15:08:51.332774 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 15:08:51.334827 (+ 2053us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:08:51.335883 (+ 1056us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:08:51.335894 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:08:51.336820 (+ 926us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:08:51.337693 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:08:51.337719 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:08:51.339051 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:08:51.339059 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:08:51.341016 (+ 1957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:08:51.341028 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:08:51.342344 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:08:51.342352 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:08:51.342527 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:08:51.343271 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:08:51.343306 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:08:51.346205 (+ 2899us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:08:51.350763 (+ 4558us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:08:51.350773 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:08:51.350796 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:08:51.351406 (+ 610us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:08:51.352017 (+ 611us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:08:51.352026 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:08:51.352032 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:08:51.352347 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:08:51.353046 (+ 699us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:08:51.353054 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:08:51.353627 (+ 573us) client_negotiation.cc:769] Sending connection context
0926 15:08:51.354030 (+ 403us) client_negotiation.cc:240] Negotiation successful
0926 15:08:51.354715 (+ 685us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":613,"thread_start_us":377,"threads_started":1}
I20260926 15:08:51.375923 13761 rpc_server.cc:307] RPC server started. Bound to: 127.12.34.194:32909
I20260926 15:08:51.375989 13895 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.34.194:32909 every 8 connection(s)
I20260926 15:08:51.377597 13761 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435318278681-12427-0/minicluster-data/ts-1/data/info.pb
I20260926 15:08:51.377981 13867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:08:51.363364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:40761 (local address 127.12.34.194:48485)
0926 15:08:51.364029 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0926 15:08:51.364065 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 15:08:51.364223 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:08:51.364745 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:08:51.364751 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:08:51.365343 (+ 592us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:08:51.365644 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:08:51.365654 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:08:51.366759 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:08:51.366764 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:08:51.368606 (+ 1842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:08:51.368617 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:08:51.369522 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:08:51.369532 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:08:51.369781 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:08:51.370569 (+ 788us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:08:51.370600 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:08:51.371719 (+ 1119us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:08:51.375307 (+ 3588us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:08:51.375312 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:08:51.375317 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:08:51.375874 (+ 557us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:08:51.376291 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:08:51.376295 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:08:51.376297 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:08:51.376406 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:08:51.377060 (+ 654us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:08:51.377064 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:08:51.377264 (+ 200us) client_negotiation.cc:769] Sending connection context
0926 15:08:51.377374 (+ 110us) client_negotiation.cc:240] Negotiation successful
0926 15:08:51.377615 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":362,"threads_started":1}
I20260926 15:08:51.378047 13645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:08:51.363212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.194:48485 (local address 127.12.34.254:40761)
0926 15:08:51.363419 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0926 15:08:51.363427 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 15:08:51.364158 (+ 731us) server_negotiation.cc:407] Connection header received
0926 15:08:51.364385 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:08:51.364393 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:08:51.364470 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:08:51.364524 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:08:51.365789 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:08:51.366634 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:08:51.368746 (+ 2112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:08:51.369269 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:08:51.369402 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:08:51.371909 (+ 2507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:08:51.371914 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:08:51.371917 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:08:51.371945 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:08:51.375176 (+ 3231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:08:51.376014 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:08:51.376019 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:08:51.376022 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:08:51.376169 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:08:51.376516 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:08:51.376520 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:08:51.376523 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:08:51.376948 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:08:51.377134 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0926 15:08:51.377505 (+ 371us) server_negotiation.cc:299] Negotiation successful
0926 15:08:51.377721 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 15:08:51.379963 13896 heartbeater.cc:347] Connected to a master server at 127.12.34.254:40761
I20260926 15:08:51.380203 13896 heartbeater.cc:464] Registering TS with master...
I20260926 15:08:51.381016 13896 heartbeater.cc:511] Master 127.12.34.254:40761 requested a full tablet report, sending...
I20260926 15:08:51.382347 13549 ts_manager.cc:194] Registered new tserver with Master: 0be73b1f9d444997ab7d9ff8ef11a74f (127.12.34.194:32909)
I20260926 15:08:51.383177 13549 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.34.194@KRBTEST.COM'} at 127.12.34.194:57103
I20260926 15:08:51.383440 12427 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8uoJGv/build/asan/bin/kudu as pid 13761
I20260926 15:08:51.383637 12427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435318278681-12427-0/minicluster-data/ts-1/wal/instance
I20260926 15:08:51.384181 12427 external_mini_cluster.cc:1428] Setting key d258f652b0a4f103257de1f3aa3fba33
WARNING: no policy specified for kudu/127.12.34.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.34.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.34.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.195.keytab.
Entry for principal kudu/127.12.34.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.195.keytab.
I20260926 15:08:51.425514 12427 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8uoJGv/build/asan/bin/kudu
/tmp/dist-test-task8uoJGv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435318278681-12427-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435318278681-12427-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435318278681-12427-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435318278681-12427-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.34.195:0
--local_ip_for_outbound_sockets=127.12.34.195
--webserver_interface=127.12.34.195
--webserver_port=0
--tserver_master_addrs=127.12.34.254:40761
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.195.keytab
--principal=kudu/127.12.34.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.34.212:46705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435318278681-12427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435318278681-12427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435318278681-12427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:08:51.739681 13900 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:08:51.740022 13900 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:08:51.740119 13900 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:08:51.747131 13900 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:08:51.747232 13900 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:08:51.747314 13900 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 15:08:51.747413 13900 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.34.195
I20260926 15:08:51.757052 13900 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.34.212:46705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435318278681-12427-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435318278681-12427-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.195.keytab
--principal=kudu/127.12.34.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.34.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435318278681-12427-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.34.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.12.34.254:40761
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.34.195
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435318278681-12427-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:08:51.758227 13900 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:08:51.759449 13900 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:08:51.773051 13908 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:08:51.773105 13906 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:08:51.773178 13905 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:08:51.773906 13900 server_base.cc:1034] running on GCE node
I20260926 15:08:51.774611 13900 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:08:51.775560 13900 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:08:51.776778 13900 hybrid_clock.cc:648] HybridClock initialized: now 1790435331776708 us; error 57 us; skew 500 ppm
I20260926 15:08:51.777132 13900 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:08:51 dist-test-slave-865v krb5kdc[13505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435331, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:08:51.780408 13900 init.cc:377] Logged in from keytab as kudu/127.12.34.195@KRBTEST.COM (short username kudu)
I20260926 15:08:51.781978 13900 webserver.cc:460] Webserver started at http://127.12.34.195:44713/ using document root <none> and password file <none>
I20260926 15:08:51.782626 13900 fs_manager.cc:359] Metadata directory not provided
I20260926 15:08:51.782711 13900 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:08:51.782955 13900 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:08:51.785727 13900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435318278681-12427-0/minicluster-data/ts-2/data/instance:
uuid: "1d71395342b847c1b2f243a4fd1be929"
format_stamp: "Formatted at 2026-09-26 15:08:51 on dist-test-slave-865v"
server_key: "0f2adce2f5983fac8ad711801b0608fe"
server_key_iv: "e41dba7131a1927353a5ece37ac4e895"
server_key_version: "encryptionkey@0"
I20260926 15:08:51.786440 13900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435318278681-12427-0/minicluster-data/ts-2/wal/instance:
uuid: "1d71395342b847c1b2f243a4fd1be929"
format_stamp: "Formatted at 2026-09-26 15:08:51 on dist-test-slave-865v"
server_key: "0f2adce2f5983fac8ad711801b0608fe"
server_key_iv: "e41dba7131a1927353a5ece37ac4e895"
server_key_version: "encryptionkey@0"
I20260926 15:08:51.791419 13900 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260926 15:08:51.794704 13915 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:08:51.795898 13900 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 15:08:51.796054 13900 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435318278681-12427-0/minicluster-data/ts-2/data,/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435318278681-12427-0/minicluster-data/ts-2/wal
uuid: "1d71395342b847c1b2f243a4fd1be929"
format_stamp: "Formatted at 2026-09-26 15:08:51 on dist-test-slave-865v"
server_key: "0f2adce2f5983fac8ad711801b0608fe"
server_key_iv: "e41dba7131a1927353a5ece37ac4e895"
server_key_version: "encryptionkey@0"
I20260926 15:08:51.796255 13900 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435318278681-12427-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435318278681-12427-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435318278681-12427-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:08:51.813936 13900 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:08:51.819320 13900 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:08:51.819628 13900 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:08:51.823282 13900 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:08:51.823446 13900 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260926 15:08:51.823601 13900 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:08:51.823679 13900 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 15:08:51 dist-test-slave-865v krb5kdc[13505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435331, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.195@KRBTEST.COM for kudu/127.12.34.254@KRBTEST.COM
I20260926 15:08:51.843443 13645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:08:51.823282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.195:50813 (local address 127.12.34.254:40761)
0926 15:08:51.823449 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0926 15:08:51.823455 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:08:51.825819 (+ 2364us) server_negotiation.cc:407] Connection header received
0926 15:08:51.827924 (+ 2105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:08:51.827930 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:08:51.828010 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:08:51.828125 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:08:51.829818 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:08:51.830583 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:08:51.832189 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:08:51.832640 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:08:51.832765 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:08:51.836239 (+ 3474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:08:51.836244 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:08:51.836247 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:08:51.836274 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:08:51.839791 (+ 3517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:08:51.840688 (+ 897us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:08:51.840694 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:08:51.840698 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:08:51.840846 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:08:51.841529 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:08:51.841536 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:08:51.841541 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:08:51.841944 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:08:51.842101 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0926 15:08:51.842843 (+ 742us) server_negotiation.cc:299] Negotiation successful
0926 15:08:51.843085 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 15:08:51.844362 13924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:08:51.823810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:40761 (local address 127.12.34.195:50813)
0926 15:08:51.825671 (+ 1861us) negotiation.cc:107] Waiting for socket to connect
0926 15:08:51.825717 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 15:08:51.827625 (+ 1908us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:08:51.828355 (+ 730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:08:51.828362 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:08:51.828943 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:08:51.829663 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:08:51.829680 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:08:51.830733 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:08:51.830739 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:08:51.832048 (+ 1309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:08:51.832059 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:08:51.832865 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:08:51.832872 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:08:51.833060 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:08:51.833863 (+ 803us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:08:51.833889 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:08:51.836049 (+ 2160us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:08:51.839957 (+ 3908us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:08:51.839969 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:08:51.839984 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:08:51.840515 (+ 531us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:08:51.841000 (+ 485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:08:51.841008 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:08:51.841015 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:08:51.841337 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:08:51.842106 (+ 769us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:08:51.842116 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:08:51.842417 (+ 301us) client_negotiation.cc:769] Sending connection context
0926 15:08:51.842666 (+ 249us) client_negotiation.cc:240] Negotiation successful
0926 15:08:51.843141 (+ 475us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1591,"thread_start_us":1339,"threads_started":1}
I20260926 15:08:51.875259 13645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:08:51.850824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.195:49111 (local address 127.12.34.254:40761)
0926 15:08:51.851025 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0926 15:08:51.851033 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 15:08:51.857686 (+ 6653us) server_negotiation.cc:407] Connection header received
0926 15:08:51.857882 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:08:51.857890 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:08:51.858006 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:08:51.858129 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:08:51.860017 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:08:51.860935 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:08:51.863536 (+ 2601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:08:51.864274 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:08:51.864392 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:08:51.867298 (+ 2906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:08:51.867308 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:08:51.867317 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:08:51.867374 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:08:51.871895 (+ 4521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:08:51.872777 (+ 882us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:08:51.872785 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:08:51.872791 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:08:51.872972 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:08:51.873415 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:08:51.873431 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:08:51.873437 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:08:51.873867 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:08:51.874001 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0926 15:08:51.874518 (+ 517us) server_negotiation.cc:299] Negotiation successful
0926 15:08:51.874873 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 15:08:51.875322 13946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:08:51.850888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:40761 (local address 127.12.34.195:49111)
0926 15:08:51.857541 (+ 6653us) negotiation.cc:107] Waiting for socket to connect
0926 15:08:51.857567 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0926 15:08:51.857739 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:08:51.858332 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:08:51.858345 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:08:51.859262 (+ 917us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:08:51.859842 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:08:51.859858 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:08:51.861151 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:08:51.861158 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:08:51.863319 (+ 2161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:08:51.863340 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:08:51.864716 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:08:51.864723 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:08:51.864952 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:08:51.865784 (+ 832us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:08:51.865816 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:08:51.867060 (+ 1244us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:08:51.872030 (+ 4970us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:08:51.872037 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:08:51.872041 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:08:51.872637 (+ 596us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:08:51.873114 (+ 477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:08:51.873121 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:08:51.873127 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:08:51.873286 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:08:51.874001 (+ 715us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:08:51.874008 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:08:51.874218 (+ 210us) client_negotiation.cc:769] Sending connection context
0926 15:08:51.874391 (+ 173us) client_negotiation.cc:240] Negotiation successful
0926 15:08:51.874769 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6450,"thread_start_us":6272,"threads_started":1}
I20260926 15:08:51.887050 13757 heartbeater.cc:503] Master 127.12.34.254:40761 was elected leader, sending a full tablet report...
I20260926 15:08:51.903049 13900 rpc_server.cc:307] RPC server started. Bound to: 127.12.34.195:35613
I20260926 15:08:51.903070 14034 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.34.195:35613 every 8 connection(s)
I20260926 15:08:51.904474 13900 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435318278681-12427-0/minicluster-data/ts-2/data/info.pb
I20260926 15:08:51.907151 14035 heartbeater.cc:347] Connected to a master server at 127.12.34.254:40761
I20260926 15:08:51.907436 14035 heartbeater.cc:464] Registering TS with master...
I20260926 15:08:51.908406 14035 heartbeater.cc:511] Master 127.12.34.254:40761 requested a full tablet report, sending...
I20260926 15:08:51.910043 13549 ts_manager.cc:194] Registered new tserver with Master: 1d71395342b847c1b2f243a4fd1be929 (127.12.34.195:35613)
I20260926 15:08:51.911032 13549 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.34.195@KRBTEST.COM'} at 127.12.34.195:50813
I20260926 15:08:51.912722 12427 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8uoJGv/build/asan/bin/kudu as pid 13900
I20260926 15:08:51.912917 12427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435318278681-12427-0/minicluster-data/ts-2/wal/instance
I20260926 15:08:51.913403 12427 external_mini_cluster.cc:1428] Setting key 2500f6c8dfb21586a0fd3baa312c22d4
I20260926 15:08:51.916095 12427 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 15:08:51 dist-test-slave-865v krb5kdc[13505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435331, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 15:08:51 dist-test-slave-865v krb5kdc[13505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435331, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.12.34.254@KRBTEST.COM
I20260926 15:08:51.963647 13645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:08:51.945657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57072 (local address 127.12.34.254:40761)
0926 15:08:51.945895 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0926 15:08:51.945903 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 15:08:51.946159 (+ 256us) server_negotiation.cc:407] Connection header received
0926 15:08:51.946369 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:08:51.946378 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:08:51.946488 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:08:51.946618 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:08:51.948026 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:08:51.949441 (+ 1415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:08:51.950744 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:08:51.951637 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:08:51.951781 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:08:51.955956 (+ 4175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:08:51.955966 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:08:51.955973 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:08:51.956018 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:08:51.960155 (+ 4137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:08:51.960769 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:08:51.960776 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:08:51.960782 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:08:51.960942 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:08:51.961457 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:08:51.961464 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:08:51.961469 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:08:51.961866 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:08:51.962148 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0926 15:08:51.962746 (+ 598us) server_negotiation.cc:299] Negotiation successful
0926 15:08:51.963120 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260926 15:08:51.968072 13549 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57072:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 15:08:51.976938 13549 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 15:08:52.029894 14048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:08:51.990328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.195:35613 (local address 127.0.0.1:56728)
0926 15:08:52.016872 (+ 26544us) negotiation.cc:107] Waiting for socket to connect
0926 15:08:52.016925 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 15:08:52.017816 (+ 891us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:08:52.020102 (+ 2286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:08:52.020110 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:08:52.020150 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:08:52.020837 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:08:52.020858 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:08:52.024111 (+ 3253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:08:52.024118 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:08:52.026457 (+ 2339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:08:52.026470 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:08:52.028745 (+ 2275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:08:52.028750 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:08:52.028917 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:08:52.028958 (+ 41us) client_negotiation.cc:769] Sending connection context
0926 15:08:52.029046 (+ 88us) client_negotiation.cc:240] Negotiation successful
0926 15:08:52.029203 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25250,"mutex_wait_us":1,"thread_start_us":28553,"threads_started":1}
I20260926 15:08:52.029994 14052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:08:51.991124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37458 (local address 127.12.34.194:32909)
0926 15:08:52.016826 (+ 25702us) server_negotiation.cc:206] Beginning negotiation
0926 15:08:52.016839 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 15:08:52.018686 (+ 1847us) server_negotiation.cc:407] Connection header received
0926 15:08:52.018960 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:08:52.018991 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:08:52.019242 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:08:52.019480 (+ 238us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:08:52.020288 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:08:52.022086 (+ 1798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:08:52.025690 (+ 3604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:08:52.027517 (+ 1827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:08:52.027647 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:08:52.027926 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0926 15:08:52.029173 (+ 1247us) server_negotiation.cc:299] Negotiation successful
0926 15:08:52.029396 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25510,"thread_start_us":25431,"threads_started":1}
I20260926 15:08:52.030887 14051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:08:51.991058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56728 (local address 127.12.34.195:35613)
0926 15:08:52.017654 (+ 26596us) server_negotiation.cc:206] Beginning negotiation
0926 15:08:52.017666 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 15:08:52.017711 (+ 45us) server_negotiation.cc:407] Connection header received
0926 15:08:52.018114 (+ 403us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:08:52.018125 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:08:52.018449 (+ 324us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:08:52.018790 (+ 341us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:08:52.021175 (+ 2385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:08:52.023940 (+ 2765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:08:52.026809 (+ 2869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:08:52.028273 (+ 1464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:08:52.028398 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:08:52.028596 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0926 15:08:52.030379 (+ 1783us) server_negotiation.cc:299] Negotiation successful
0926 15:08:52.030565 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26359,"thread_start_us":26679,"threads_started":1}
I20260926 15:08:52.030812 14049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:08:51.990483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.194:32909 (local address 127.0.0.1:37458)
0926 15:08:52.018556 (+ 28073us) negotiation.cc:107] Waiting for socket to connect
0926 15:08:52.018577 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 15:08:52.018734 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:08:52.019513 (+ 779us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:08:52.019523 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:08:52.019574 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:08:52.020058 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:08:52.020072 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:08:52.022289 (+ 2217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:08:52.022294 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:08:52.025492 (+ 3198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:08:52.025510 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:08:52.028139 (+ 2629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:08:52.028147 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:08:52.028458 (+ 311us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:08:52.028535 (+ 77us) client_negotiation.cc:769] Sending connection context
0926 15:08:52.028695 (+ 160us) client_negotiation.cc:240] Negotiation successful
0926 15:08:52.028922 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27906,"mutex_wait_us":6,"thread_start_us":25564,"threads_started":1}
I20260926 15:08:52.032760 14053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:08:51.991596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46920 (local address 127.12.34.193:39853)
0926 15:08:52.016065 (+ 24469us) server_negotiation.cc:206] Beginning negotiation
0926 15:08:52.016076 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 15:08:52.022929 (+ 6853us) server_negotiation.cc:407] Connection header received
0926 15:08:52.023219 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:08:52.023227 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:08:52.023488 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:08:52.023715 (+ 227us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:08:52.024454 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:08:52.026143 (+ 1689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:08:52.028891 (+ 2748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:08:52.030591 (+ 1700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:08:52.030702 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:08:52.031805 (+ 1103us) server_negotiation.cc:1035] Waiting for connection context
0926 15:08:52.032317 (+ 512us) server_negotiation.cc:299] Negotiation successful
0926 15:08:52.032461 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24281,"thread_start_us":24473,"threads_started":1}
I20260926 15:08:52.032763 14050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:08:51.991088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.193:39853 (local address 127.0.0.1:46920)
0926 15:08:52.022822 (+ 31734us) negotiation.cc:107] Waiting for socket to connect
0926 15:08:52.022839 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 15:08:52.022954 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:08:52.023702 (+ 748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:08:52.023706 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:08:52.023732 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:08:52.024236 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:08:52.024249 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:08:52.026334 (+ 2085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:08:52.026338 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:08:52.028728 (+ 2390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:08:52.028739 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:08:52.031982 (+ 3243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:08:52.031988 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:08:52.032143 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:08:52.032164 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 15:08:52.032268 (+ 104us) client_negotiation.cc:240] Negotiation successful
0926 15:08:52.032389 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31624,"mutex_wait_us":4,"thread_start_us":32241,"threads_started":1}
I20260926 15:08:52.035610 13687 tablet_service.cc:1467] Processing CreateTablet for tablet 2c1a2f05665c4181a771778f1670f556 (DEFAULT_TABLE table=test-table [id=f8719aa0f1f54fa3be095ccb33613946]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:08:52.035669 13970 tablet_service.cc:1467] Processing CreateTablet for tablet 2c1a2f05665c4181a771778f1670f556 (DEFAULT_TABLE table=test-table [id=f8719aa0f1f54fa3be095ccb33613946]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:08:52.036746 13687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c1a2f05665c4181a771778f1670f556. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:08:52.037498 13970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c1a2f05665c4181a771778f1670f556. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:08:52.040582 13826 tablet_service.cc:1467] Processing CreateTablet for tablet 2c1a2f05665c4181a771778f1670f556 (DEFAULT_TABLE table=test-table [id=f8719aa0f1f54fa3be095ccb33613946]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:08:52.041913 13826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c1a2f05665c4181a771778f1670f556. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:08:52.058709 14054 tablet_bootstrap.cc:492] T 2c1a2f05665c4181a771778f1670f556 P a74c4495c62e440bbf55cd02a879911c: Bootstrap starting.
I20260926 15:08:52.060395 14055 tablet_bootstrap.cc:492] T 2c1a2f05665c4181a771778f1670f556 P 1d71395342b847c1b2f243a4fd1be929: Bootstrap starting.
I20260926 15:08:52.060626 14056 tablet_bootstrap.cc:492] T 2c1a2f05665c4181a771778f1670f556 P 0be73b1f9d444997ab7d9ff8ef11a74f: Bootstrap starting.
I20260926 15:08:52.064724 14056 tablet_bootstrap.cc:654] T 2c1a2f05665c4181a771778f1670f556 P 0be73b1f9d444997ab7d9ff8ef11a74f: Neither blocks nor log segments found. Creating new log.
I20260926 15:08:52.064987 14054 tablet_bootstrap.cc:654] T 2c1a2f05665c4181a771778f1670f556 P a74c4495c62e440bbf55cd02a879911c: Neither blocks nor log segments found. Creating new log.
I20260926 15:08:52.066007 14056 log.cc:824] T 2c1a2f05665c4181a771778f1670f556 P 0be73b1f9d444997ab7d9ff8ef11a74f: Log is configured to *not* fsync() on all Append() calls
I20260926 15:08:52.066135 14054 log.cc:824] T 2c1a2f05665c4181a771778f1670f556 P a74c4495c62e440bbf55cd02a879911c: Log is configured to *not* fsync() on all Append() calls
I20260926 15:08:52.068670 14055 tablet_bootstrap.cc:654] T 2c1a2f05665c4181a771778f1670f556 P 1d71395342b847c1b2f243a4fd1be929: Neither blocks nor log segments found. Creating new log.
I20260926 15:08:52.070152 14055 log.cc:824] T 2c1a2f05665c4181a771778f1670f556 P 1d71395342b847c1b2f243a4fd1be929: Log is configured to *not* fsync() on all Append() calls
I20260926 15:08:52.075824 14054 tablet_bootstrap.cc:492] T 2c1a2f05665c4181a771778f1670f556 P a74c4495c62e440bbf55cd02a879911c: No bootstrap required, opened a new log
I20260926 15:08:52.075826 14056 tablet_bootstrap.cc:492] T 2c1a2f05665c4181a771778f1670f556 P 0be73b1f9d444997ab7d9ff8ef11a74f: No bootstrap required, opened a new log
I20260926 15:08:52.076108 14056 ts_tablet_manager.cc:1397] T 2c1a2f05665c4181a771778f1670f556 P 0be73b1f9d444997ab7d9ff8ef11a74f: Time spent bootstrapping tablet: real 0.016s user 0.009s sys 0.004s
I20260926 15:08:52.076108 14054 ts_tablet_manager.cc:1397] T 2c1a2f05665c4181a771778f1670f556 P a74c4495c62e440bbf55cd02a879911c: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.007s
I20260926 15:08:52.079661 14055 tablet_bootstrap.cc:492] T 2c1a2f05665c4181a771778f1670f556 P 1d71395342b847c1b2f243a4fd1be929: No bootstrap required, opened a new log
I20260926 15:08:52.080078 14055 ts_tablet_manager.cc:1397] T 2c1a2f05665c4181a771778f1670f556 P 1d71395342b847c1b2f243a4fd1be929: Time spent bootstrapping tablet: real 0.020s user 0.014s sys 0.000s
I20260926 15:08:52.087424 14056 raft_consensus.cc:348] T 2c1a2f05665c4181a771778f1670f556 P 0be73b1f9d444997ab7d9ff8ef11a74f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d71395342b847c1b2f243a4fd1be929" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 35613 } } peers { permanent_uuid: "0be73b1f9d444997ab7d9ff8ef11a74f" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 32909 } } peers { permanent_uuid: "a74c4495c62e440bbf55cd02a879911c" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 39853 } }
I20260926 15:08:52.087423 14054 raft_consensus.cc:348] T 2c1a2f05665c4181a771778f1670f556 P a74c4495c62e440bbf55cd02a879911c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d71395342b847c1b2f243a4fd1be929" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 35613 } } peers { permanent_uuid: "0be73b1f9d444997ab7d9ff8ef11a74f" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 32909 } } peers { permanent_uuid: "a74c4495c62e440bbf55cd02a879911c" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 39853 } }
I20260926 15:08:52.087875 14054 raft_consensus.cc:374] T 2c1a2f05665c4181a771778f1670f556 P a74c4495c62e440bbf55cd02a879911c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:08:52.088016 14054 raft_consensus.cc:729] T 2c1a2f05665c4181a771778f1670f556 P a74c4495c62e440bbf55cd02a879911c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a74c4495c62e440bbf55cd02a879911c, State: Initialized, Role: FOLLOWER
I20260926 15:08:52.088034 14056 raft_consensus.cc:374] T 2c1a2f05665c4181a771778f1670f556 P 0be73b1f9d444997ab7d9ff8ef11a74f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:08:52.088131 14056 raft_consensus.cc:729] T 2c1a2f05665c4181a771778f1670f556 P 0be73b1f9d444997ab7d9ff8ef11a74f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0be73b1f9d444997ab7d9ff8ef11a74f, State: Initialized, Role: FOLLOWER
I20260926 15:08:52.088869 14054 consensus_queue.cc:260] T 2c1a2f05665c4181a771778f1670f556 P a74c4495c62e440bbf55cd02a879911c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d71395342b847c1b2f243a4fd1be929" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 35613 } } peers { permanent_uuid: "0be73b1f9d444997ab7d9ff8ef11a74f" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 32909 } } peers { permanent_uuid: "a74c4495c62e440bbf55cd02a879911c" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 39853 } }
I20260926 15:08:52.088869 14056 consensus_queue.cc:260] T 2c1a2f05665c4181a771778f1670f556 P 0be73b1f9d444997ab7d9ff8ef11a74f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d71395342b847c1b2f243a4fd1be929" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 35613 } } peers { permanent_uuid: "0be73b1f9d444997ab7d9ff8ef11a74f" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 32909 } } peers { permanent_uuid: "a74c4495c62e440bbf55cd02a879911c" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 39853 } }
I20260926 15:08:52.094636 14056 ts_tablet_manager.cc:1428] T 2c1a2f05665c4181a771778f1670f556 P 0be73b1f9d444997ab7d9ff8ef11a74f: Time spent starting tablet: real 0.018s user 0.013s sys 0.002s
I20260926 15:08:52.094805 14054 ts_tablet_manager.cc:1428] T 2c1a2f05665c4181a771778f1670f556 P a74c4495c62e440bbf55cd02a879911c: Time spent starting tablet: real 0.019s user 0.017s sys 0.000s
I20260926 15:08:52.095270 14055 raft_consensus.cc:348] T 2c1a2f05665c4181a771778f1670f556 P 1d71395342b847c1b2f243a4fd1be929 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d71395342b847c1b2f243a4fd1be929" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 35613 } } peers { permanent_uuid: "0be73b1f9d444997ab7d9ff8ef11a74f" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 32909 } } peers { permanent_uuid: "a74c4495c62e440bbf55cd02a879911c" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 39853 } }
I20260926 15:08:52.095680 13896 heartbeater.cc:503] Master 127.12.34.254:40761 was elected leader, sending a full tablet report...
I20260926 15:08:52.095731 14055 raft_consensus.cc:374] T 2c1a2f05665c4181a771778f1670f556 P 1d71395342b847c1b2f243a4fd1be929 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:08:52.095826 14055 raft_consensus.cc:729] T 2c1a2f05665c4181a771778f1670f556 P 1d71395342b847c1b2f243a4fd1be929 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d71395342b847c1b2f243a4fd1be929, State: Initialized, Role: FOLLOWER
I20260926 15:08:52.096433 14055 consensus_queue.cc:260] T 2c1a2f05665c4181a771778f1670f556 P 1d71395342b847c1b2f243a4fd1be929 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d71395342b847c1b2f243a4fd1be929" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 35613 } } peers { permanent_uuid: "0be73b1f9d444997ab7d9ff8ef11a74f" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 32909 } } peers { permanent_uuid: "a74c4495c62e440bbf55cd02a879911c" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 39853 } }
I20260926 15:08:52.102253 14035 heartbeater.cc:503] Master 127.12.34.254:40761 was elected leader, sending a full tablet report...
I20260926 15:08:52.102777 14055 ts_tablet_manager.cc:1428] T 2c1a2f05665c4181a771778f1670f556 P 1d71395342b847c1b2f243a4fd1be929: Time spent starting tablet: real 0.023s user 0.016s sys 0.004s
W20260926 15:08:52.130136 13897 tablet.cc:2369] T 2c1a2f05665c4181a771778f1670f556 P 0be73b1f9d444997ab7d9ff8ef11a74f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 15:08:52.131137 13758 tablet.cc:2369] T 2c1a2f05665c4181a771778f1670f556 P a74c4495c62e440bbf55cd02a879911c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 15:08:52.156929 14062 raft_consensus.cc:482] T 2c1a2f05665c4181a771778f1670f556 P 1d71395342b847c1b2f243a4fd1be929 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260926 15:08:52.157541 14036 tablet.cc:2369] T 2c1a2f05665c4181a771778f1670f556 P 1d71395342b847c1b2f243a4fd1be929: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 15:08:52.157394 14062 raft_consensus.cc:504] T 2c1a2f05665c4181a771778f1670f556 P 1d71395342b847c1b2f243a4fd1be929 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d71395342b847c1b2f243a4fd1be929" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 35613 } } peers { permanent_uuid: "0be73b1f9d444997ab7d9ff8ef11a74f" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 32909 } } peers { permanent_uuid: "a74c4495c62e440bbf55cd02a879911c" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 39853 } }
I20260926 15:08:52.159443 14062 leader_election.cc:288] T 2c1a2f05665c4181a771778f1670f556 P 1d71395342b847c1b2f243a4fd1be929 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0be73b1f9d444997ab7d9ff8ef11a74f (127.12.34.194:32909), a74c4495c62e440bbf55cd02a879911c (127.12.34.193:39853)
I20260926 15:08:52.165774 14053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:08:52.159923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.195:59729 (local address 127.12.34.193:39853)
0926 15:08:52.160139 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0926 15:08:52.160146 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:08:52.160366 (+ 220us) server_negotiation.cc:407] Connection header received
0926 15:08:52.160595 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:08:52.160601 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:08:52.160670 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:08:52.160803 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:08:52.161266 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:08:52.161992 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:08:52.163546 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:08:52.164299 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:08:52.164477 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:08:52.164551 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 15:08:52.165385 (+ 834us) server_negotiation.cc:299] Negotiation successful
0926 15:08:52.165516 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 15:08:52.165861 13924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:08:52.159967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.193:39853 (local address 127.12.34.195:59729)
0926 15:08:52.160267 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0926 15:08:52.160282 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 15:08:52.160449 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:08:52.160829 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:08:52.160841 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:08:52.160869 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:08:52.161134 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:08:52.161145 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:08:52.162138 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:08:52.162143 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:08:52.163385 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:08:52.163397 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:08:52.164466 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:08:52.164472 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:08:52.164932 (+ 460us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:08:52.164977 (+ 45us) client_negotiation.cc:769] Sending connection context
0926 15:08:52.165254 (+ 277us) client_negotiation.cc:240] Negotiation successful
0926 15:08:52.165502 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"spinlock_wait_cycles":12032}
I20260926 15:08:52.165868 14052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:08:52.159949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.195:40481 (local address 127.12.34.194:32909)
0926 15:08:52.160161 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0926 15:08:52.160169 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 15:08:52.160863 (+ 694us) server_negotiation.cc:407] Connection header received
0926 15:08:52.161129 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:08:52.161134 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:08:52.161201 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:08:52.161297 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:08:52.161787 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:08:52.162513 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:08:52.164123 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:08:52.164906 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:08:52.165082 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:08:52.165180 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 15:08:52.165497 (+ 317us) server_negotiation.cc:299] Negotiation successful
0926 15:08:52.165616 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260926 15:08:52.166460 14063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:08:52.160003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.194:32909 (local address 127.12.34.195:40481)
0926 15:08:52.160754 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0926 15:08:52.160771 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 15:08:52.160950 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:08:52.161309 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:08:52.161314 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:08:52.161336 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:08:52.161625 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:08:52.161633 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:08:52.162645 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:08:52.162652 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:08:52.163971 (+ 1319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:08:52.163983 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:08:52.165187 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:08:52.165194 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:08:52.165329 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:08:52.165354 (+ 25us) client_negotiation.cc:769] Sending connection context
0926 15:08:52.165466 (+ 112us) client_negotiation.cc:240] Negotiation successful
0926 15:08:52.165589 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":651,"spinlock_wait_cycles":3456,"thread_start_us":463,"threads_started":1}
I20260926 15:08:52.167244 13709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c1a2f05665c4181a771778f1670f556" candidate_uuid: "1d71395342b847c1b2f243a4fd1be929" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a74c4495c62e440bbf55cd02a879911c" is_pre_election: true
I20260926 15:08:52.167414 13846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c1a2f05665c4181a771778f1670f556" candidate_uuid: "1d71395342b847c1b2f243a4fd1be929" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0be73b1f9d444997ab7d9ff8ef11a74f" is_pre_election: true
I20260926 15:08:52.167861 13709 raft_consensus.cc:2454] T 2c1a2f05665c4181a771778f1670f556 P a74c4495c62e440bbf55cd02a879911c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d71395342b847c1b2f243a4fd1be929 in term 0.
I20260926 15:08:52.167860 13846 raft_consensus.cc:2454] T 2c1a2f05665c4181a771778f1670f556 P 0be73b1f9d444997ab7d9ff8ef11a74f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d71395342b847c1b2f243a4fd1be929 in term 0.
I20260926 15:08:52.168560 13917 leader_election.cc:302] T 2c1a2f05665c4181a771778f1670f556 P 1d71395342b847c1b2f243a4fd1be929 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0be73b1f9d444997ab7d9ff8ef11a74f, 1d71395342b847c1b2f243a4fd1be929; no voters:
I20260926 15:08:52.168944 14062 raft_consensus.cc:2781] T 2c1a2f05665c4181a771778f1670f556 P 1d71395342b847c1b2f243a4fd1be929 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 15:08:52.169117 14062 raft_consensus.cc:482] T 2c1a2f05665c4181a771778f1670f556 P 1d71395342b847c1b2f243a4fd1be929 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 15:08:52.169245 14062 raft_consensus.cc:3037] T 2c1a2f05665c4181a771778f1670f556 P 1d71395342b847c1b2f243a4fd1be929 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:08:52.171357 14062 raft_consensus.cc:504] T 2c1a2f05665c4181a771778f1670f556 P 1d71395342b847c1b2f243a4fd1be929 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d71395342b847c1b2f243a4fd1be929" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 35613 } } peers { permanent_uuid: "0be73b1f9d444997ab7d9ff8ef11a74f" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 32909 } } peers { permanent_uuid: "a74c4495c62e440bbf55cd02a879911c" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 39853 } }
I20260926 15:08:52.171983 14062 leader_election.cc:288] T 2c1a2f05665c4181a771778f1670f556 P 1d71395342b847c1b2f243a4fd1be929 [CANDIDATE]: Term 1 election: Requested vote from peers 0be73b1f9d444997ab7d9ff8ef11a74f (127.12.34.194:32909), a74c4495c62e440bbf55cd02a879911c (127.12.34.193:39853)
I20260926 15:08:52.172438 13709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c1a2f05665c4181a771778f1670f556" candidate_uuid: "1d71395342b847c1b2f243a4fd1be929" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a74c4495c62e440bbf55cd02a879911c"
I20260926 15:08:52.172441 13846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c1a2f05665c4181a771778f1670f556" candidate_uuid: "1d71395342b847c1b2f243a4fd1be929" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0be73b1f9d444997ab7d9ff8ef11a74f"
I20260926 15:08:52.172640 13846 raft_consensus.cc:3037] T 2c1a2f05665c4181a771778f1670f556 P 0be73b1f9d444997ab7d9ff8ef11a74f [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:08:52.172624 13709 raft_consensus.cc:3037] T 2c1a2f05665c4181a771778f1670f556 P a74c4495c62e440bbf55cd02a879911c [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:08:52.174875 13709 raft_consensus.cc:2454] T 2c1a2f05665c4181a771778f1670f556 P a74c4495c62e440bbf55cd02a879911c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d71395342b847c1b2f243a4fd1be929 in term 1.
I20260926 15:08:52.174913 13846 raft_consensus.cc:2454] T 2c1a2f05665c4181a771778f1670f556 P 0be73b1f9d444997ab7d9ff8ef11a74f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d71395342b847c1b2f243a4fd1be929 in term 1.
I20260926 15:08:52.175247 13916 leader_election.cc:302] T 2c1a2f05665c4181a771778f1670f556 P 1d71395342b847c1b2f243a4fd1be929 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d71395342b847c1b2f243a4fd1be929, a74c4495c62e440bbf55cd02a879911c; no voters:
I20260926 15:08:52.175586 14062 raft_consensus.cc:2781] T 2c1a2f05665c4181a771778f1670f556 P 1d71395342b847c1b2f243a4fd1be929 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:08:52.176030 14062 raft_consensus.cc:686] T 2c1a2f05665c4181a771778f1670f556 P 1d71395342b847c1b2f243a4fd1be929 [term 1 LEADER]: Becoming Leader. State: Replica: 1d71395342b847c1b2f243a4fd1be929, State: Running, Role: LEADER
I20260926 15:08:52.176550 14062 consensus_queue.cc:237] T 2c1a2f05665c4181a771778f1670f556 P 1d71395342b847c1b2f243a4fd1be929 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d71395342b847c1b2f243a4fd1be929" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 35613 } } peers { permanent_uuid: "0be73b1f9d444997ab7d9ff8ef11a74f" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 32909 } } peers { permanent_uuid: "a74c4495c62e440bbf55cd02a879911c" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 39853 } }
I20260926 15:08:52.181485 13549 catalog_manager.cc:5495] T 2c1a2f05665c4181a771778f1670f556 P 1d71395342b847c1b2f243a4fd1be929 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d71395342b847c1b2f243a4fd1be929 (127.12.34.195). New cstate: current_term: 1 leader_uuid: "1d71395342b847c1b2f243a4fd1be929" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d71395342b847c1b2f243a4fd1be929" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 35613 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0be73b1f9d444997ab7d9ff8ef11a74f" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 32909 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a74c4495c62e440bbf55cd02a879911c" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 39853 } health_report { overall_health: UNKNOWN } } }
I20260926 15:08:52.202726 14051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:08:52.197139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56738 (local address 127.12.34.195:35613)
0926 15:08:52.197378 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0926 15:08:52.197385 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:08:52.197535 (+ 150us) server_negotiation.cc:407] Connection header received
0926 15:08:52.197747 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:08:52.197750 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:08:52.197832 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:08:52.197948 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 15:08:52.198323 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:08:52.199060 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:08:52.200494 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:08:52.200933 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:08:52.201028 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:08:52.201583 (+ 555us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 15:08:52.202080 (+ 497us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 15:08:52.202191 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0926 15:08:52.202368 (+ 177us) server_negotiation.cc:299] Negotiation successful
0926 15:08:52.202431 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 15:08:52.211493 13846 raft_consensus.cc:1261] T 2c1a2f05665c4181a771778f1670f556 P 0be73b1f9d444997ab7d9ff8ef11a74f [term 1 FOLLOWER]: Refusing update from remote peer 1d71395342b847c1b2f243a4fd1be929: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:08:52.211493 13709 raft_consensus.cc:1261] T 2c1a2f05665c4181a771778f1670f556 P a74c4495c62e440bbf55cd02a879911c [term 1 FOLLOWER]: Refusing update from remote peer 1d71395342b847c1b2f243a4fd1be929: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:08:52.212539 14064 consensus_queue.cc:1035] T 2c1a2f05665c4181a771778f1670f556 P 1d71395342b847c1b2f243a4fd1be929 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a74c4495c62e440bbf55cd02a879911c" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 39853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:08:52.212951 14062 consensus_queue.cc:1035] T 2c1a2f05665c4181a771778f1670f556 P 1d71395342b847c1b2f243a4fd1be929 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0be73b1f9d444997ab7d9ff8ef11a74f" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 32909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:08:52.230458 14067 mvcc.cc:205] Tried to move back new op lower bound from 7333623120727203840 to 7333623120599175168. Current Snapshot: MvccSnapshot[applied={T|T < 7333623120727203840}]
I20260926 15:08:52.249033 13550 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57072:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 15:08:52.249360 13550 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57072:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 15:08:52.253764 13550 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e855e2d27ca44f7fbc65d11d9bc0b942: Sending DeleteTablet for 3 replicas of tablet 2c1a2f05665c4181a771778f1670f556
I20260926 15:08:52.254843 13970 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c1a2f05665c4181a771778f1670f556 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:08:52 UTC) from {username='kudu', principal='kudu/127.12.34.254@KRBTEST.COM'} at 127.0.0.1:56728
I20260926 15:08:52.255025 13826 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c1a2f05665c4181a771778f1670f556 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:08:52 UTC) from {username='kudu', principal='kudu/127.12.34.254@KRBTEST.COM'} at 127.0.0.1:37458
I20260926 15:08:52.255033 13687 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c1a2f05665c4181a771778f1670f556 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:08:52 UTC) from {username='kudu', principal='kudu/127.12.34.254@KRBTEST.COM'} at 127.0.0.1:46920
I20260926 15:08:52.256561 13550 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57072:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 15:08:52.257614 13550 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 15:08:52.266338 13686 tablet_service.cc:1467] Processing CreateTablet for tablet 25250331d3e74105b581ff202922a87a (DEFAULT_TABLE table=test-table [id=ef314a1a433d49fdbd6bb53718703d76]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:08:52.266614 13969 tablet_service.cc:1467] Processing CreateTablet for tablet 25250331d3e74105b581ff202922a87a (DEFAULT_TABLE table=test-table [id=ef314a1a433d49fdbd6bb53718703d76]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:08:52.266945 13686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25250331d3e74105b581ff202922a87a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:08:52.267158 13969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25250331d3e74105b581ff202922a87a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:08:52.271438 13825 tablet_service.cc:1467] Processing CreateTablet for tablet 25250331d3e74105b581ff202922a87a (DEFAULT_TABLE table=test-table [id=ef314a1a433d49fdbd6bb53718703d76]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:08:52.272094 13825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25250331d3e74105b581ff202922a87a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:08:52.273854 14076 tablet_replica.cc:330] T 2c1a2f05665c4181a771778f1670f556 P 1d71395342b847c1b2f243a4fd1be929: stopping tablet replica
I20260926 15:08:52.274292 14076 raft_consensus.cc:2229] T 2c1a2f05665c4181a771778f1670f556 P 1d71395342b847c1b2f243a4fd1be929 [term 1 LEADER]: Raft consensus shutting down.
I20260926 15:08:52.274942 14076 raft_consensus.cc:2258] T 2c1a2f05665c4181a771778f1670f556 P 1d71395342b847c1b2f243a4fd1be929 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:08:52.275280 14055 tablet_bootstrap.cc:492] T 25250331d3e74105b581ff202922a87a P 1d71395342b847c1b2f243a4fd1be929: Bootstrap starting.
I20260926 15:08:52.276867 14055 tablet_bootstrap.cc:654] T 25250331d3e74105b581ff202922a87a P 1d71395342b847c1b2f243a4fd1be929: Neither blocks nor log segments found. Creating new log.
I20260926 15:08:52.288343 14055 tablet_bootstrap.cc:492] T 25250331d3e74105b581ff202922a87a P 1d71395342b847c1b2f243a4fd1be929: No bootstrap required, opened a new log
I20260926 15:08:52.288599 14055 ts_tablet_manager.cc:1397] T 25250331d3e74105b581ff202922a87a P 1d71395342b847c1b2f243a4fd1be929: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.008s
I20260926 15:08:52.288652 14056 tablet_bootstrap.cc:492] T 25250331d3e74105b581ff202922a87a P 0be73b1f9d444997ab7d9ff8ef11a74f: Bootstrap starting.
I20260926 15:08:52.289582 14077 tablet_replica.cc:330] T 2c1a2f05665c4181a771778f1670f556 P 0be73b1f9d444997ab7d9ff8ef11a74f: stopping tablet replica
I20260926 15:08:52.289525 14055 raft_consensus.cc:348] T 25250331d3e74105b581ff202922a87a P 1d71395342b847c1b2f243a4fd1be929 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a74c4495c62e440bbf55cd02a879911c" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 39853 } } peers { permanent_uuid: "0be73b1f9d444997ab7d9ff8ef11a74f" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 32909 } } peers { permanent_uuid: "1d71395342b847c1b2f243a4fd1be929" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 35613 } }
I20260926 15:08:52.289882 14055 raft_consensus.cc:374] T 25250331d3e74105b581ff202922a87a P 1d71395342b847c1b2f243a4fd1be929 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:08:52.290035 14055 raft_consensus.cc:729] T 25250331d3e74105b581ff202922a87a P 1d71395342b847c1b2f243a4fd1be929 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d71395342b847c1b2f243a4fd1be929, State: Initialized, Role: FOLLOWER
I20260926 15:08:52.290058 14077 raft_consensus.cc:2229] T 2c1a2f05665c4181a771778f1670f556 P 0be73b1f9d444997ab7d9ff8ef11a74f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 15:08:52.290488 14056 tablet_bootstrap.cc:654] T 25250331d3e74105b581ff202922a87a P 0be73b1f9d444997ab7d9ff8ef11a74f: Neither blocks nor log segments found. Creating new log.
I20260926 15:08:52.290582 14077 raft_consensus.cc:2258] T 2c1a2f05665c4181a771778f1670f556 P 0be73b1f9d444997ab7d9ff8ef11a74f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:08:52.290426 14055 consensus_queue.cc:260] T 25250331d3e74105b581ff202922a87a P 1d71395342b847c1b2f243a4fd1be929 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a74c4495c62e440bbf55cd02a879911c" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 39853 } } peers { permanent_uuid: "0be73b1f9d444997ab7d9ff8ef11a74f" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 32909 } } peers { permanent_uuid: "1d71395342b847c1b2f243a4fd1be929" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 35613 } }
I20260926 15:08:52.291074 14076 ts_tablet_manager.cc:1905] T 2c1a2f05665c4181a771778f1670f556 P 1d71395342b847c1b2f243a4fd1be929: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:08:52.291328 14078 tablet_replica.cc:330] T 2c1a2f05665c4181a771778f1670f556 P a74c4495c62e440bbf55cd02a879911c: stopping tablet replica
I20260926 15:08:52.291813 14078 raft_consensus.cc:2229] T 2c1a2f05665c4181a771778f1670f556 P a74c4495c62e440bbf55cd02a879911c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 15:08:52.292320 14078 raft_consensus.cc:2258] T 2c1a2f05665c4181a771778f1670f556 P a74c4495c62e440bbf55cd02a879911c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:08:52.293917 14054 tablet_bootstrap.cc:492] T 25250331d3e74105b581ff202922a87a P a74c4495c62e440bbf55cd02a879911c: Bootstrap starting.
I20260926 15:08:52.294643 14078 ts_tablet_manager.cc:1905] T 2c1a2f05665c4181a771778f1670f556 P a74c4495c62e440bbf55cd02a879911c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:08:52.295094 14055 ts_tablet_manager.cc:1428] T 25250331d3e74105b581ff202922a87a P 1d71395342b847c1b2f243a4fd1be929: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20260926 15:08:52.295547 14054 tablet_bootstrap.cc:654] T 25250331d3e74105b581ff202922a87a P a74c4495c62e440bbf55cd02a879911c: Neither blocks nor log segments found. Creating new log.
I20260926 15:08:52.300727 14054 tablet_bootstrap.cc:492] T 25250331d3e74105b581ff202922a87a P a74c4495c62e440bbf55cd02a879911c: No bootstrap required, opened a new log
I20260926 15:08:52.300860 14054 ts_tablet_manager.cc:1397] T 25250331d3e74105b581ff202922a87a P a74c4495c62e440bbf55cd02a879911c: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260926 15:08:52.301314 14077 ts_tablet_manager.cc:1905] T 2c1a2f05665c4181a771778f1670f556 P 0be73b1f9d444997ab7d9ff8ef11a74f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:08:52.301614 14054 raft_consensus.cc:348] T 25250331d3e74105b581ff202922a87a P a74c4495c62e440bbf55cd02a879911c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a74c4495c62e440bbf55cd02a879911c" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 39853 } } peers { permanent_uuid: "0be73b1f9d444997ab7d9ff8ef11a74f" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 32909 } } peers { permanent_uuid: "1d71395342b847c1b2f243a4fd1be929" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 35613 } }
I20260926 15:08:52.301831 14054 raft_consensus.cc:374] T 25250331d3e74105b581ff202922a87a P a74c4495c62e440bbf55cd02a879911c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:08:52.301926 14054 raft_consensus.cc:729] T 25250331d3e74105b581ff202922a87a P a74c4495c62e440bbf55cd02a879911c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a74c4495c62e440bbf55cd02a879911c, State: Initialized, Role: FOLLOWER
I20260926 15:08:52.302174 14054 consensus_queue.cc:260] T 25250331d3e74105b581ff202922a87a P a74c4495c62e440bbf55cd02a879911c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a74c4495c62e440bbf55cd02a879911c" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 39853 } } peers { permanent_uuid: "0be73b1f9d444997ab7d9ff8ef11a74f" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 32909 } } peers { permanent_uuid: "1d71395342b847c1b2f243a4fd1be929" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 35613 } }
I20260926 15:08:52.302804 14054 ts_tablet_manager.cc:1428] T 25250331d3e74105b581ff202922a87a P a74c4495c62e440bbf55cd02a879911c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 15:08:52.304744 14076 ts_tablet_manager.cc:1918] T 2c1a2f05665c4181a771778f1670f556 P 1d71395342b847c1b2f243a4fd1be929: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 15:08:52.304905 14076 log.cc:1196] T 2c1a2f05665c4181a771778f1670f556 P 1d71395342b847c1b2f243a4fd1be929: Deleting WAL directory at /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435318278681-12427-0/minicluster-data/ts-2/wal/wals/2c1a2f05665c4181a771778f1670f556
I20260926 15:08:52.305344 14056 tablet_bootstrap.cc:492] T 25250331d3e74105b581ff202922a87a P 0be73b1f9d444997ab7d9ff8ef11a74f: No bootstrap required, opened a new log
I20260926 15:08:52.305399 14076 ts_tablet_manager.cc:1939] T 2c1a2f05665c4181a771778f1670f556 P 1d71395342b847c1b2f243a4fd1be929: Deleting consensus metadata
I20260926 15:08:52.305491 14056 ts_tablet_manager.cc:1397] T 25250331d3e74105b581ff202922a87a P 0be73b1f9d444997ab7d9ff8ef11a74f: Time spent bootstrapping tablet: real 0.017s user 0.012s sys 0.000s
I20260926 15:08:52.306344 14056 raft_consensus.cc:348] T 25250331d3e74105b581ff202922a87a P 0be73b1f9d444997ab7d9ff8ef11a74f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a74c4495c62e440bbf55cd02a879911c" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 39853 } } peers { permanent_uuid: "0be73b1f9d444997ab7d9ff8ef11a74f" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 32909 } } peers { permanent_uuid: "1d71395342b847c1b2f243a4fd1be929" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 35613 } }
I20260926 15:08:52.306562 14056 raft_consensus.cc:374] T 25250331d3e74105b581ff202922a87a P 0be73b1f9d444997ab7d9ff8ef11a74f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:08:52.306661 14056 raft_consensus.cc:729] T 25250331d3e74105b581ff202922a87a P 0be73b1f9d444997ab7d9ff8ef11a74f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0be73b1f9d444997ab7d9ff8ef11a74f, State: Initialized, Role: FOLLOWER
I20260926 15:08:52.306943 14056 consensus_queue.cc:260] T 25250331d3e74105b581ff202922a87a P 0be73b1f9d444997ab7d9ff8ef11a74f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a74c4495c62e440bbf55cd02a879911c" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 39853 } } peers { permanent_uuid: "0be73b1f9d444997ab7d9ff8ef11a74f" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 32909 } } peers { permanent_uuid: "1d71395342b847c1b2f243a4fd1be929" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 35613 } }
I20260926 15:08:52.307066 13536 catalog_manager.cc:4841] TS 1d71395342b847c1b2f243a4fd1be929 (127.12.34.195:35613): tablet 2c1a2f05665c4181a771778f1670f556 (table test-table [id=f8719aa0f1f54fa3be095ccb33613946]) successfully deleted
I20260926 15:08:52.307623 14056 ts_tablet_manager.cc:1428] T 25250331d3e74105b581ff202922a87a P 0be73b1f9d444997ab7d9ff8ef11a74f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 15:08:52.311738 14077 ts_tablet_manager.cc:1918] T 2c1a2f05665c4181a771778f1670f556 P 0be73b1f9d444997ab7d9ff8ef11a74f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 15:08:52.311898 14077 log.cc:1196] T 2c1a2f05665c4181a771778f1670f556 P 0be73b1f9d444997ab7d9ff8ef11a74f: Deleting WAL directory at /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435318278681-12427-0/minicluster-data/ts-1/wal/wals/2c1a2f05665c4181a771778f1670f556
I20260926 15:08:52.312443 14077 ts_tablet_manager.cc:1939] T 2c1a2f05665c4181a771778f1670f556 P 0be73b1f9d444997ab7d9ff8ef11a74f: Deleting consensus metadata
I20260926 15:08:52.314615 13535 catalog_manager.cc:4841] TS 0be73b1f9d444997ab7d9ff8ef11a74f (127.12.34.194:32909): tablet 2c1a2f05665c4181a771778f1670f556 (table test-table [id=f8719aa0f1f54fa3be095ccb33613946]) successfully deleted
I20260926 15:08:52.317047 14078 ts_tablet_manager.cc:1918] T 2c1a2f05665c4181a771778f1670f556 P a74c4495c62e440bbf55cd02a879911c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 15:08:52.317303 14078 log.cc:1196] T 2c1a2f05665c4181a771778f1670f556 P a74c4495c62e440bbf55cd02a879911c: Deleting WAL directory at /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435318278681-12427-0/minicluster-data/ts-0/wal/wals/2c1a2f05665c4181a771778f1670f556
I20260926 15:08:52.317890 14078 ts_tablet_manager.cc:1939] T 2c1a2f05665c4181a771778f1670f556 P a74c4495c62e440bbf55cd02a879911c: Deleting consensus metadata
I20260926 15:08:52.319993 13534 catalog_manager.cc:4841] TS a74c4495c62e440bbf55cd02a879911c (127.12.34.193:39853): tablet 2c1a2f05665c4181a771778f1670f556 (table test-table [id=f8719aa0f1f54fa3be095ccb33613946]) successfully deleted
I20260926 15:08:52.502947 14062 raft_consensus.cc:482] T 25250331d3e74105b581ff202922a87a P 1d71395342b847c1b2f243a4fd1be929 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 15:08:52.503190 14062 raft_consensus.cc:504] T 25250331d3e74105b581ff202922a87a P 1d71395342b847c1b2f243a4fd1be929 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a74c4495c62e440bbf55cd02a879911c" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 39853 } } peers { permanent_uuid: "0be73b1f9d444997ab7d9ff8ef11a74f" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 32909 } } peers { permanent_uuid: "1d71395342b847c1b2f243a4fd1be929" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 35613 } }
I20260926 15:08:52.503875 14062 leader_election.cc:288] T 25250331d3e74105b581ff202922a87a P 1d71395342b847c1b2f243a4fd1be929 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a74c4495c62e440bbf55cd02a879911c (127.12.34.193:39853), 0be73b1f9d444997ab7d9ff8ef11a74f (127.12.34.194:32909)
I20260926 15:08:52.504441 13846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25250331d3e74105b581ff202922a87a" candidate_uuid: "1d71395342b847c1b2f243a4fd1be929" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0be73b1f9d444997ab7d9ff8ef11a74f" is_pre_election: true
I20260926 15:08:52.504447 13709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25250331d3e74105b581ff202922a87a" candidate_uuid: "1d71395342b847c1b2f243a4fd1be929" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a74c4495c62e440bbf55cd02a879911c" is_pre_election: true
I20260926 15:08:52.504691 13846 raft_consensus.cc:2454] T 25250331d3e74105b581ff202922a87a P 0be73b1f9d444997ab7d9ff8ef11a74f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d71395342b847c1b2f243a4fd1be929 in term 0.
I20260926 15:08:52.504736 13709 raft_consensus.cc:2454] T 25250331d3e74105b581ff202922a87a P a74c4495c62e440bbf55cd02a879911c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d71395342b847c1b2f243a4fd1be929 in term 0.
I20260926 15:08:52.505188 13917 leader_election.cc:302] T 25250331d3e74105b581ff202922a87a P 1d71395342b847c1b2f243a4fd1be929 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0be73b1f9d444997ab7d9ff8ef11a74f, 1d71395342b847c1b2f243a4fd1be929; no voters:
I20260926 15:08:52.505457 14062 raft_consensus.cc:2781] T 25250331d3e74105b581ff202922a87a P 1d71395342b847c1b2f243a4fd1be929 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 15:08:52.505590 14062 raft_consensus.cc:482] T 25250331d3e74105b581ff202922a87a P 1d71395342b847c1b2f243a4fd1be929 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 15:08:52.505712 14062 raft_consensus.cc:3037] T 25250331d3e74105b581ff202922a87a P 1d71395342b847c1b2f243a4fd1be929 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:08:52.508455 14062 raft_consensus.cc:504] T 25250331d3e74105b581ff202922a87a P 1d71395342b847c1b2f243a4fd1be929 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a74c4495c62e440bbf55cd02a879911c" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 39853 } } peers { permanent_uuid: "0be73b1f9d444997ab7d9ff8ef11a74f" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 32909 } } peers { permanent_uuid: "1d71395342b847c1b2f243a4fd1be929" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 35613 } }
I20260926 15:08:52.509037 14062 leader_election.cc:288] T 25250331d3e74105b581ff202922a87a P 1d71395342b847c1b2f243a4fd1be929 [CANDIDATE]: Term 1 election: Requested vote from peers a74c4495c62e440bbf55cd02a879911c (127.12.34.193:39853), 0be73b1f9d444997ab7d9ff8ef11a74f (127.12.34.194:32909)
I20260926 15:08:52.509554 13709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25250331d3e74105b581ff202922a87a" candidate_uuid: "1d71395342b847c1b2f243a4fd1be929" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a74c4495c62e440bbf55cd02a879911c"
I20260926 15:08:52.509747 13709 raft_consensus.cc:3037] T 25250331d3e74105b581ff202922a87a P a74c4495c62e440bbf55cd02a879911c [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:08:52.511969 13709 raft_consensus.cc:2454] T 25250331d3e74105b581ff202922a87a P a74c4495c62e440bbf55cd02a879911c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d71395342b847c1b2f243a4fd1be929 in term 1.
I20260926 15:08:52.512367 13916 leader_election.cc:302] T 25250331d3e74105b581ff202922a87a P 1d71395342b847c1b2f243a4fd1be929 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d71395342b847c1b2f243a4fd1be929, a74c4495c62e440bbf55cd02a879911c; no voters:
I20260926 15:08:52.512380 13846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25250331d3e74105b581ff202922a87a" candidate_uuid: "1d71395342b847c1b2f243a4fd1be929" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0be73b1f9d444997ab7d9ff8ef11a74f"
I20260926 15:08:52.512573 13846 raft_consensus.cc:3037] T 25250331d3e74105b581ff202922a87a P 0be73b1f9d444997ab7d9ff8ef11a74f [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:08:52.512676 14062 raft_consensus.cc:2781] T 25250331d3e74105b581ff202922a87a P 1d71395342b847c1b2f243a4fd1be929 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:08:52.512812 14062 raft_consensus.cc:686] T 25250331d3e74105b581ff202922a87a P 1d71395342b847c1b2f243a4fd1be929 [term 1 LEADER]: Becoming Leader. State: Replica: 1d71395342b847c1b2f243a4fd1be929, State: Running, Role: LEADER
I20260926 15:08:52.513125 14062 consensus_queue.cc:237] T 25250331d3e74105b581ff202922a87a P 1d71395342b847c1b2f243a4fd1be929 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a74c4495c62e440bbf55cd02a879911c" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 39853 } } peers { permanent_uuid: "0be73b1f9d444997ab7d9ff8ef11a74f" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 32909 } } peers { permanent_uuid: "1d71395342b847c1b2f243a4fd1be929" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 35613 } }
I20260926 15:08:52.515928 13846 raft_consensus.cc:2454] T 25250331d3e74105b581ff202922a87a P 0be73b1f9d444997ab7d9ff8ef11a74f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d71395342b847c1b2f243a4fd1be929 in term 1.
I20260926 15:08:52.518325 13550 catalog_manager.cc:5495] T 25250331d3e74105b581ff202922a87a P 1d71395342b847c1b2f243a4fd1be929 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d71395342b847c1b2f243a4fd1be929 (127.12.34.195). New cstate: current_term: 1 leader_uuid: "1d71395342b847c1b2f243a4fd1be929" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a74c4495c62e440bbf55cd02a879911c" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 39853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0be73b1f9d444997ab7d9ff8ef11a74f" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 32909 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d71395342b847c1b2f243a4fd1be929" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 35613 } health_report { overall_health: HEALTHY } } }
Sep 26 15:08:52 dist-test-slave-865v krb5kdc[13505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435330, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.254@KRBTEST.COM for kudu/127.12.34.254@KRBTEST.COM
I20260926 15:08:52.619387 14088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:08:52.603360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:40761 (local address 127.0.0.1:57084)
0926 15:08:52.604235 (+ 875us) negotiation.cc:107] Waiting for socket to connect
0926 15:08:52.604252 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 15:08:52.604367 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:08:52.604736 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:08:52.604739 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:08:52.605408 (+ 669us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:08:52.605620 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:08:52.605628 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:08:52.607186 (+ 1558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:08:52.607191 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:08:52.608329 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:08:52.608340 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:08:52.609301 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:08:52.609306 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:08:52.609445 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:08:52.610143 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:08:52.610160 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:08:52.612751 (+ 2591us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:08:52.617118 (+ 4367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:08:52.617135 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:08:52.617140 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:08:52.617532 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:08:52.617915 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:08:52.617917 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:08:52.617920 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:08:52.618050 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:08:52.618646 (+ 596us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:08:52.618652 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:08:52.618785 (+ 133us) client_negotiation.cc:769] Sending connection context
0926 15:08:52.618918 (+ 133us) client_negotiation.cc:240] Negotiation successful
0926 15:08:52.619118 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":749,"thread_start_us":607,"threads_started":1}
I20260926 15:08:52.619683 14089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:08:52.603398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57084 (local address 127.12.34.254:40761)
0926 15:08:52.604005 (+ 607us) server_negotiation.cc:206] Beginning negotiation
0926 15:08:52.604015 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 15:08:52.604341 (+ 326us) server_negotiation.cc:407] Connection header received
0926 15:08:52.604486 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:08:52.604491 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:08:52.604571 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:08:52.604703 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:08:52.605769 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:08:52.606999 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:08:52.608483 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:08:52.609115 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:08:52.609241 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:08:52.613356 (+ 4115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:08:52.613364 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:08:52.613369 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:08:52.613405 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:08:52.616958 (+ 3553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:08:52.617655 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:08:52.617661 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:08:52.617665 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:08:52.617806 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:08:52.618154 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:08:52.618159 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:08:52.618162 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:08:52.618519 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:08:52.618661 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0926 15:08:52.619042 (+ 381us) server_negotiation.cc:299] Negotiation successful
0926 15:08:52.619257 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":456,"thread_start_us":306,"threads_started":1}
I20260926 15:08:52.625005 13550 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.12.34.254@KRBTEST.COM'} at 127.0.0.1:57084:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 15:08:52.626237 13550 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 15:08:52.633468 13686 tablet_service.cc:1467] Processing CreateTablet for tablet 7a1a19b0ca2a49c49b5b1626c007d639 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=15abffca274c40f3b15da5001e3c1303]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 15:08:52.633980 13686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a1a19b0ca2a49c49b5b1626c007d639. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:08:52.633846 13969 tablet_service.cc:1467] Processing CreateTablet for tablet 7a1a19b0ca2a49c49b5b1626c007d639 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=15abffca274c40f3b15da5001e3c1303]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 15:08:52.633929 13826 tablet_service.cc:1467] Processing CreateTablet for tablet 7a1a19b0ca2a49c49b5b1626c007d639 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=15abffca274c40f3b15da5001e3c1303]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 15:08:52.634337 13969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a1a19b0ca2a49c49b5b1626c007d639. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:08:52.634407 13826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a1a19b0ca2a49c49b5b1626c007d639. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:08:52.641147 14056 tablet_bootstrap.cc:492] T 7a1a19b0ca2a49c49b5b1626c007d639 P 0be73b1f9d444997ab7d9ff8ef11a74f: Bootstrap starting.
I20260926 15:08:52.641287 14054 tablet_bootstrap.cc:492] T 7a1a19b0ca2a49c49b5b1626c007d639 P a74c4495c62e440bbf55cd02a879911c: Bootstrap starting.
I20260926 15:08:52.642781 14054 tablet_bootstrap.cc:654] T 7a1a19b0ca2a49c49b5b1626c007d639 P a74c4495c62e440bbf55cd02a879911c: Neither blocks nor log segments found. Creating new log.
I20260926 15:08:52.643298 14056 tablet_bootstrap.cc:654] T 7a1a19b0ca2a49c49b5b1626c007d639 P 0be73b1f9d444997ab7d9ff8ef11a74f: Neither blocks nor log segments found. Creating new log.
I20260926 15:08:52.644775 14054 tablet_bootstrap.cc:492] T 7a1a19b0ca2a49c49b5b1626c007d639 P a74c4495c62e440bbf55cd02a879911c: No bootstrap required, opened a new log
I20260926 15:08:52.644961 14054 ts_tablet_manager.cc:1397] T 7a1a19b0ca2a49c49b5b1626c007d639 P a74c4495c62e440bbf55cd02a879911c: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260926 15:08:52.645720 14054 raft_consensus.cc:348] T 7a1a19b0ca2a49c49b5b1626c007d639 P a74c4495c62e440bbf55cd02a879911c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a74c4495c62e440bbf55cd02a879911c" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 39853 } } peers { permanent_uuid: "0be73b1f9d444997ab7d9ff8ef11a74f" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 32909 } } peers { permanent_uuid: "1d71395342b847c1b2f243a4fd1be929" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 35613 } }
I20260926 15:08:52.645987 14054 raft_consensus.cc:374] T 7a1a19b0ca2a49c49b5b1626c007d639 P a74c4495c62e440bbf55cd02a879911c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:08:52.646076 14054 raft_consensus.cc:729] T 7a1a19b0ca2a49c49b5b1626c007d639 P a74c4495c62e440bbf55cd02a879911c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a74c4495c62e440bbf55cd02a879911c, State: Initialized, Role: FOLLOWER
I20260926 15:08:52.646328 14054 consensus_queue.cc:260] T 7a1a19b0ca2a49c49b5b1626c007d639 P a74c4495c62e440bbf55cd02a879911c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a74c4495c62e440bbf55cd02a879911c" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 39853 } } peers { permanent_uuid: "0be73b1f9d444997ab7d9ff8ef11a74f" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 32909 } } peers { permanent_uuid: "1d71395342b847c1b2f243a4fd1be929" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 35613 } }
I20260926 15:08:52.646901 14054 ts_tablet_manager.cc:1428] T 7a1a19b0ca2a49c49b5b1626c007d639 P a74c4495c62e440bbf55cd02a879911c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 15:08:52.647351 14055 tablet_bootstrap.cc:492] T 7a1a19b0ca2a49c49b5b1626c007d639 P 1d71395342b847c1b2f243a4fd1be929: Bootstrap starting.
I20260926 15:08:52.648618 14055 tablet_bootstrap.cc:654] T 7a1a19b0ca2a49c49b5b1626c007d639 P 1d71395342b847c1b2f243a4fd1be929: Neither blocks nor log segments found. Creating new log.
I20260926 15:08:52.650074 14060 tablet_replica.cc:439] T 7a1a19b0ca2a49c49b5b1626c007d639 P a74c4495c62e440bbf55cd02a879911c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a74c4495c62e440bbf55cd02a879911c" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 39853 } } peers { permanent_uuid: "0be73b1f9d444997ab7d9ff8ef11a74f" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 32909 } } peers { permanent_uuid: "1d71395342b847c1b2f243a4fd1be929" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 35613 } } }
I20260926 15:08:52.650379 14060 tablet_replica.cc:442] T 7a1a19b0ca2a49c49b5b1626c007d639 P a74c4495c62e440bbf55cd02a879911c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 15:08:52.652268 14056 tablet_bootstrap.cc:492] T 7a1a19b0ca2a49c49b5b1626c007d639 P 0be73b1f9d444997ab7d9ff8ef11a74f: No bootstrap required, opened a new log
I20260926 15:08:52.652277 14055 tablet_bootstrap.cc:492] T 7a1a19b0ca2a49c49b5b1626c007d639 P 1d71395342b847c1b2f243a4fd1be929: No bootstrap required, opened a new log
I20260926 15:08:52.652419 14056 ts_tablet_manager.cc:1397] T 7a1a19b0ca2a49c49b5b1626c007d639 P 0be73b1f9d444997ab7d9ff8ef11a74f: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.003s
I20260926 15:08:52.652428 14055 ts_tablet_manager.cc:1397] T 7a1a19b0ca2a49c49b5b1626c007d639 P 1d71395342b847c1b2f243a4fd1be929: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260926 15:08:52.653220 14056 raft_consensus.cc:348] T 7a1a19b0ca2a49c49b5b1626c007d639 P 0be73b1f9d444997ab7d9ff8ef11a74f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a74c4495c62e440bbf55cd02a879911c" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 39853 } } peers { permanent_uuid: "0be73b1f9d444997ab7d9ff8ef11a74f" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 32909 } } peers { permanent_uuid: "1d71395342b847c1b2f243a4fd1be929" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 35613 } }
I20260926 15:08:52.653213 14055 raft_consensus.cc:348] T 7a1a19b0ca2a49c49b5b1626c007d639 P 1d71395342b847c1b2f243a4fd1be929 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a74c4495c62e440bbf55cd02a879911c" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 39853 } } peers { permanent_uuid: "0be73b1f9d444997ab7d9ff8ef11a74f" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 32909 } } peers { permanent_uuid: "1d71395342b847c1b2f243a4fd1be929" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 35613 } }
I20260926 15:08:52.653436 14056 raft_consensus.cc:374] T 7a1a19b0ca2a49c49b5b1626c007d639 P 0be73b1f9d444997ab7d9ff8ef11a74f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:08:52.653453 14055 raft_consensus.cc:374] T 7a1a19b0ca2a49c49b5b1626c007d639 P 1d71395342b847c1b2f243a4fd1be929 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:08:52.653507 14056 raft_consensus.cc:729] T 7a1a19b0ca2a49c49b5b1626c007d639 P 0be73b1f9d444997ab7d9ff8ef11a74f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0be73b1f9d444997ab7d9ff8ef11a74f, State: Initialized, Role: FOLLOWER
I20260926 15:08:52.653529 14055 raft_consensus.cc:729] T 7a1a19b0ca2a49c49b5b1626c007d639 P 1d71395342b847c1b2f243a4fd1be929 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d71395342b847c1b2f243a4fd1be929, State: Initialized, Role: FOLLOWER
I20260926 15:08:52.653759 14055 consensus_queue.cc:260] T 7a1a19b0ca2a49c49b5b1626c007d639 P 1d71395342b847c1b2f243a4fd1be929 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a74c4495c62e440bbf55cd02a879911c" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 39853 } } peers { permanent_uuid: "0be73b1f9d444997ab7d9ff8ef11a74f" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 32909 } } peers { permanent_uuid: "1d71395342b847c1b2f243a4fd1be929" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 35613 } }
I20260926 15:08:52.653785 14056 consensus_queue.cc:260] T 7a1a19b0ca2a49c49b5b1626c007d639 P 0be73b1f9d444997ab7d9ff8ef11a74f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a74c4495c62e440bbf55cd02a879911c" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 39853 } } peers { permanent_uuid: "0be73b1f9d444997ab7d9ff8ef11a74f" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 32909 } } peers { permanent_uuid: "1d71395342b847c1b2f243a4fd1be929" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 35613 } }
I20260926 15:08:52.654414 14055 ts_tablet_manager.cc:1428] T 7a1a19b0ca2a49c49b5b1626c007d639 P 1d71395342b847c1b2f243a4fd1be929: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260926 15:08:52.654213 14061 tablet_replica.cc:439] T 7a1a19b0ca2a49c49b5b1626c007d639 P 0be73b1f9d444997ab7d9ff8ef11a74f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a74c4495c62e440bbf55cd02a879911c" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 39853 } } peers { permanent_uuid: "0be73b1f9d444997ab7d9ff8ef11a74f" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 32909 } } peers { permanent_uuid: "1d71395342b847c1b2f243a4fd1be929" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 35613 } } }
I20260926 15:08:52.654237 14062 tablet_replica.cc:439] T 7a1a19b0ca2a49c49b5b1626c007d639 P 1d71395342b847c1b2f243a4fd1be929: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a74c4495c62e440bbf55cd02a879911c" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 39853 } } peers { permanent_uuid: "0be73b1f9d444997ab7d9ff8ef11a74f" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 32909 } } peers { permanent_uuid: "1d71395342b847c1b2f243a4fd1be929" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 35613 } } }
I20260926 15:08:52.654541 14061 tablet_replica.cc:442] T 7a1a19b0ca2a49c49b5b1626c007d639 P 0be73b1f9d444997ab7d9ff8ef11a74f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 15:08:52.654635 14062 tablet_replica.cc:442] T 7a1a19b0ca2a49c49b5b1626c007d639 P 1d71395342b847c1b2f243a4fd1be929: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 15:08:52.655414 14056 ts_tablet_manager.cc:1428] T 7a1a19b0ca2a49c49b5b1626c007d639 P 0be73b1f9d444997ab7d9ff8ef11a74f: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260926 15:08:52.755407 14061 raft_consensus.cc:482] T 7a1a19b0ca2a49c49b5b1626c007d639 P 0be73b1f9d444997ab7d9ff8ef11a74f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 15:08:52.755681 14061 raft_consensus.cc:504] T 7a1a19b0ca2a49c49b5b1626c007d639 P 0be73b1f9d444997ab7d9ff8ef11a74f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a74c4495c62e440bbf55cd02a879911c" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 39853 } } peers { permanent_uuid: "0be73b1f9d444997ab7d9ff8ef11a74f" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 32909 } } peers { permanent_uuid: "1d71395342b847c1b2f243a4fd1be929" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 35613 } }
I20260926 15:08:52.757242 14061 leader_election.cc:288] T 7a1a19b0ca2a49c49b5b1626c007d639 P 0be73b1f9d444997ab7d9ff8ef11a74f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a74c4495c62e440bbf55cd02a879911c (127.12.34.193:39853), 1d71395342b847c1b2f243a4fd1be929 (127.12.34.195:35613)
I20260926 15:08:52.775786 14094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:08:52.757711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.194:42349 (local address 127.12.34.195:35613)
0926 15:08:52.770803 (+ 13092us) server_negotiation.cc:206] Beginning negotiation
0926 15:08:52.770807 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:08:52.770823 (+ 16us) server_negotiation.cc:407] Connection header received
0926 15:08:52.770883 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:08:52.770887 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:08:52.770942 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:08:52.771081 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:08:52.771635 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:08:52.772280 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:08:52.774045 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:08:52.774784 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:08:52.774962 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:08:52.775039 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 15:08:52.775311 (+ 272us) server_negotiation.cc:299] Negotiation successful
0926 15:08:52.775454 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12999,"thread_start_us":12912,"threads_started":1}
I20260926 15:08:52.775833 14095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:08:52.757696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.195:35613 (local address 127.12.34.194:42349)
0926 15:08:52.770503 (+ 12807us) negotiation.cc:107] Waiting for socket to connect
0926 15:08:52.770520 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 15:08:52.770610 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:08:52.771106 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:08:52.771111 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:08:52.771139 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:08:52.771470 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:08:52.771479 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:08:52.772420 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:08:52.772428 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:08:52.773881 (+ 1453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:08:52.773899 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:08:52.774949 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:08:52.774957 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:08:52.775153 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:08:52.775177 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 15:08:52.775285 (+ 108us) client_negotiation.cc:240] Negotiation successful
0926 15:08:52.775429 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12708,"thread_start_us":12572,"threads_started":1}
I20260926 15:08:52.776466 14093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:08:52.757465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.193:39853 (local address 127.12.34.194:43721)
0926 15:08:52.770189 (+ 12724us) negotiation.cc:107] Waiting for socket to connect
0926 15:08:52.770221 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0926 15:08:52.770386 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:08:52.770824 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:08:52.770829 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:08:52.770852 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:08:52.771108 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:08:52.771117 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:08:52.772142 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:08:52.772148 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:08:52.773739 (+ 1591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:08:52.773754 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:08:52.775175 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:08:52.775183 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:08:52.775446 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:08:52.775535 (+ 89us) client_negotiation.cc:769] Sending connection context
0926 15:08:52.775684 (+ 149us) client_negotiation.cc:240] Negotiation successful
0926 15:08:52.775901 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12612,"mutex_wait_us":58,"thread_start_us":12513,"threads_started":1}
I20260926 15:08:52.777139 14096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:08:52.757691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.194:43721 (local address 127.12.34.193:39853)
0926 15:08:52.769474 (+ 11783us) server_negotiation.cc:206] Beginning negotiation
0926 15:08:52.769484 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 15:08:52.770504 (+ 1020us) server_negotiation.cc:407] Connection header received
0926 15:08:52.770597 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:08:52.770602 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:08:52.770682 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:08:52.770808 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:08:52.771247 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:08:52.771983 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:08:52.773905 (+ 1922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:08:52.774700 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:08:52.775162 (+ 462us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:08:52.775265 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 15:08:52.775850 (+ 585us) server_negotiation.cc:299] Negotiation successful
0926 15:08:52.776590 (+ 740us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11584,"thread_start_us":12011,"threads_started":1}
I20260926 15:08:52.777281 13990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a1a19b0ca2a49c49b5b1626c007d639" candidate_uuid: "0be73b1f9d444997ab7d9ff8ef11a74f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d71395342b847c1b2f243a4fd1be929" is_pre_election: true
I20260926 15:08:52.777717 13990 raft_consensus.cc:2454] T 7a1a19b0ca2a49c49b5b1626c007d639 P 1d71395342b847c1b2f243a4fd1be929 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0be73b1f9d444997ab7d9ff8ef11a74f in term 0.
I20260926 15:08:52.778030 13709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a1a19b0ca2a49c49b5b1626c007d639" candidate_uuid: "0be73b1f9d444997ab7d9ff8ef11a74f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a74c4495c62e440bbf55cd02a879911c" is_pre_election: true
I20260926 15:08:52.778326 13709 raft_consensus.cc:2454] T 7a1a19b0ca2a49c49b5b1626c007d639 P a74c4495c62e440bbf55cd02a879911c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0be73b1f9d444997ab7d9ff8ef11a74f in term 0.
I20260926 15:08:52.778316 13779 leader_election.cc:302] T 7a1a19b0ca2a49c49b5b1626c007d639 P 0be73b1f9d444997ab7d9ff8ef11a74f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0be73b1f9d444997ab7d9ff8ef11a74f, 1d71395342b847c1b2f243a4fd1be929; no voters:
I20260926 15:08:52.778739 14061 raft_consensus.cc:2781] T 7a1a19b0ca2a49c49b5b1626c007d639 P 0be73b1f9d444997ab7d9ff8ef11a74f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 15:08:52.778846 14061 raft_consensus.cc:482] T 7a1a19b0ca2a49c49b5b1626c007d639 P 0be73b1f9d444997ab7d9ff8ef11a74f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 15:08:52.778986 14061 raft_consensus.cc:3037] T 7a1a19b0ca2a49c49b5b1626c007d639 P 0be73b1f9d444997ab7d9ff8ef11a74f [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:08:52.781045 14061 raft_consensus.cc:504] T 7a1a19b0ca2a49c49b5b1626c007d639 P 0be73b1f9d444997ab7d9ff8ef11a74f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a74c4495c62e440bbf55cd02a879911c" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 39853 } } peers { permanent_uuid: "0be73b1f9d444997ab7d9ff8ef11a74f" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 32909 } } peers { permanent_uuid: "1d71395342b847c1b2f243a4fd1be929" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 35613 } }
I20260926 15:08:52.781766 14061 leader_election.cc:288] T 7a1a19b0ca2a49c49b5b1626c007d639 P 0be73b1f9d444997ab7d9ff8ef11a74f [CANDIDATE]: Term 1 election: Requested vote from peers a74c4495c62e440bbf55cd02a879911c (127.12.34.193:39853), 1d71395342b847c1b2f243a4fd1be929 (127.12.34.195:35613)
I20260926 15:08:52.782207 13709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a1a19b0ca2a49c49b5b1626c007d639" candidate_uuid: "0be73b1f9d444997ab7d9ff8ef11a74f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a74c4495c62e440bbf55cd02a879911c"
I20260926 15:08:52.782238 13990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a1a19b0ca2a49c49b5b1626c007d639" candidate_uuid: "0be73b1f9d444997ab7d9ff8ef11a74f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d71395342b847c1b2f243a4fd1be929"
I20260926 15:08:52.782438 13990 raft_consensus.cc:3037] T 7a1a19b0ca2a49c49b5b1626c007d639 P 1d71395342b847c1b2f243a4fd1be929 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:08:52.782455 13709 raft_consensus.cc:3037] T 7a1a19b0ca2a49c49b5b1626c007d639 P a74c4495c62e440bbf55cd02a879911c [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:08:52.785403 13709 raft_consensus.cc:2454] T 7a1a19b0ca2a49c49b5b1626c007d639 P a74c4495c62e440bbf55cd02a879911c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0be73b1f9d444997ab7d9ff8ef11a74f in term 1.
I20260926 15:08:52.785404 13990 raft_consensus.cc:2454] T 7a1a19b0ca2a49c49b5b1626c007d639 P 1d71395342b847c1b2f243a4fd1be929 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0be73b1f9d444997ab7d9ff8ef11a74f in term 1.
I20260926 15:08:52.785789 13779 leader_election.cc:302] T 7a1a19b0ca2a49c49b5b1626c007d639 P 0be73b1f9d444997ab7d9ff8ef11a74f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0be73b1f9d444997ab7d9ff8ef11a74f, 1d71395342b847c1b2f243a4fd1be929; no voters:
I20260926 15:08:52.786083 14061 raft_consensus.cc:2781] T 7a1a19b0ca2a49c49b5b1626c007d639 P 0be73b1f9d444997ab7d9ff8ef11a74f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:08:52.786485 14061 raft_consensus.cc:686] T 7a1a19b0ca2a49c49b5b1626c007d639 P 0be73b1f9d444997ab7d9ff8ef11a74f [term 1 LEADER]: Becoming Leader. State: Replica: 0be73b1f9d444997ab7d9ff8ef11a74f, State: Running, Role: LEADER
I20260926 15:08:52.786855 14061 consensus_queue.cc:237] T 7a1a19b0ca2a49c49b5b1626c007d639 P 0be73b1f9d444997ab7d9ff8ef11a74f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a74c4495c62e440bbf55cd02a879911c" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 39853 } } peers { permanent_uuid: "0be73b1f9d444997ab7d9ff8ef11a74f" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 32909 } } peers { permanent_uuid: "1d71395342b847c1b2f243a4fd1be929" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 35613 } }
I20260926 15:08:52.788735 14097 tablet_replica.cc:439] T 7a1a19b0ca2a49c49b5b1626c007d639 P 0be73b1f9d444997ab7d9ff8ef11a74f: TxnStatusTablet state changed. Reason: New leader 0be73b1f9d444997ab7d9ff8ef11a74f. Latest consensus state: current_term: 1 leader_uuid: "0be73b1f9d444997ab7d9ff8ef11a74f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a74c4495c62e440bbf55cd02a879911c" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 39853 } } peers { permanent_uuid: "0be73b1f9d444997ab7d9ff8ef11a74f" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 32909 } } peers { permanent_uuid: "1d71395342b847c1b2f243a4fd1be929" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 35613 } } }
I20260926 15:08:52.789156 14097 tablet_replica.cc:442] T 7a1a19b0ca2a49c49b5b1626c007d639 P 0be73b1f9d444997ab7d9ff8ef11a74f: This TxnStatusTablet replica's current role is: LEADER
I20260926 15:08:52.789860 14099 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 15:08:52.790087 14099 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 15:08:52.790470 13550 catalog_manager.cc:5495] T 7a1a19b0ca2a49c49b5b1626c007d639 P 0be73b1f9d444997ab7d9ff8ef11a74f reported cstate change: term changed from 0 to 1, leader changed from <none> to 0be73b1f9d444997ab7d9ff8ef11a74f (127.12.34.194). New cstate: current_term: 1 leader_uuid: "0be73b1f9d444997ab7d9ff8ef11a74f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a74c4495c62e440bbf55cd02a879911c" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 39853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0be73b1f9d444997ab7d9ff8ef11a74f" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 32909 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1d71395342b847c1b2f243a4fd1be929" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 35613 } health_report { overall_health: UNKNOWN } } }
I20260926 15:08:52.814656 14102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:08:52.806773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37464 (local address 127.12.34.194:32909)
0926 15:08:52.808339 (+ 1566us) server_negotiation.cc:206] Beginning negotiation
0926 15:08:52.808344 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:08:52.808366 (+ 22us) server_negotiation.cc:407] Connection header received
0926 15:08:52.808442 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:08:52.808445 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:08:52.808515 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:08:52.808637 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 15:08:52.809314 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:08:52.810126 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:08:52.812181 (+ 2055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:08:52.812737 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:08:52.812834 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:08:52.813418 (+ 584us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 15:08:52.814022 (+ 604us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 15:08:52.814151 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0926 15:08:52.814301 (+ 150us) server_negotiation.cc:299] Negotiation successful
0926 15:08:52.814361 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1419,"thread_start_us":1175,"threads_started":1}
I20260926 15:08:52.814769 14088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:08:52.806775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.194:32909 (local address 127.0.0.1:37464)
0926 15:08:52.807171 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0926 15:08:52.807193 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 15:08:52.807321 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:08:52.808699 (+ 1378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:08:52.808705 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:08:52.808719 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 15:08:52.808906 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:08:52.808918 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:08:52.810294 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:08:52.810299 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:08:52.812045 (+ 1746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:08:52.812061 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:08:52.812886 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:08:52.812892 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:08:52.813125 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:08:52.813194 (+ 69us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 15:08:52.814151 (+ 957us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 15:08:52.814179 (+ 28us) client_negotiation.cc:769] Sending connection context
0926 15:08:52.814276 (+ 97us) client_negotiation.cc:240] Negotiation successful
0926 15:08:52.814423 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225}
I20260926 15:08:52.818956 13709 raft_consensus.cc:1261] T 7a1a19b0ca2a49c49b5b1626c007d639 P a74c4495c62e440bbf55cd02a879911c [term 1 FOLLOWER]: Refusing update from remote peer 0be73b1f9d444997ab7d9ff8ef11a74f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:08:52.819639 14097 consensus_queue.cc:1035] T 7a1a19b0ca2a49c49b5b1626c007d639 P 0be73b1f9d444997ab7d9ff8ef11a74f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a74c4495c62e440bbf55cd02a879911c" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 39853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:08:52.819815 13990 raft_consensus.cc:1261] T 7a1a19b0ca2a49c49b5b1626c007d639 P 1d71395342b847c1b2f243a4fd1be929 [term 1 FOLLOWER]: Refusing update from remote peer 0be73b1f9d444997ab7d9ff8ef11a74f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:08:52.820727 14097 consensus_queue.cc:1035] T 7a1a19b0ca2a49c49b5b1626c007d639 P 0be73b1f9d444997ab7d9ff8ef11a74f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d71395342b847c1b2f243a4fd1be929" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 35613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:08:52.826995 14062 tablet_replica.cc:439] T 7a1a19b0ca2a49c49b5b1626c007d639 P 1d71395342b847c1b2f243a4fd1be929: TxnStatusTablet state changed. Reason: New leader 0be73b1f9d444997ab7d9ff8ef11a74f. Latest consensus state: current_term: 1 leader_uuid: "0be73b1f9d444997ab7d9ff8ef11a74f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a74c4495c62e440bbf55cd02a879911c" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 39853 } } peers { permanent_uuid: "0be73b1f9d444997ab7d9ff8ef11a74f" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 32909 } } peers { permanent_uuid: "1d71395342b847c1b2f243a4fd1be929" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 35613 } } }
I20260926 15:08:52.827860 14062 tablet_replica.cc:442] T 7a1a19b0ca2a49c49b5b1626c007d639 P 1d71395342b847c1b2f243a4fd1be929: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 15:08:52.831573 14060 tablet_replica.cc:439] T 7a1a19b0ca2a49c49b5b1626c007d639 P a74c4495c62e440bbf55cd02a879911c: TxnStatusTablet state changed. Reason: New leader 0be73b1f9d444997ab7d9ff8ef11a74f. Latest consensus state: current_term: 1 leader_uuid: "0be73b1f9d444997ab7d9ff8ef11a74f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a74c4495c62e440bbf55cd02a879911c" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 39853 } } peers { permanent_uuid: "0be73b1f9d444997ab7d9ff8ef11a74f" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 32909 } } peers { permanent_uuid: "1d71395342b847c1b2f243a4fd1be929" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 35613 } } }
I20260926 15:08:52.831878 14060 tablet_replica.cc:442] T 7a1a19b0ca2a49c49b5b1626c007d639 P a74c4495c62e440bbf55cd02a879911c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 15:08:52.834355 14061 tablet_replica.cc:439] T 7a1a19b0ca2a49c49b5b1626c007d639 P 0be73b1f9d444997ab7d9ff8ef11a74f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0be73b1f9d444997ab7d9ff8ef11a74f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a74c4495c62e440bbf55cd02a879911c" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 39853 } } peers { permanent_uuid: "0be73b1f9d444997ab7d9ff8ef11a74f" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 32909 } } peers { permanent_uuid: "1d71395342b847c1b2f243a4fd1be929" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 35613 } } }
I20260926 15:08:52.834645 14061 tablet_replica.cc:442] T 7a1a19b0ca2a49c49b5b1626c007d639 P 0be73b1f9d444997ab7d9ff8ef11a74f: This TxnStatusTablet replica's current role is: LEADER
I20260926 15:08:52.834824 14061 tablet_replica.cc:439] T 7a1a19b0ca2a49c49b5b1626c007d639 P 0be73b1f9d444997ab7d9ff8ef11a74f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0be73b1f9d444997ab7d9ff8ef11a74f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a74c4495c62e440bbf55cd02a879911c" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 39853 } } peers { permanent_uuid: "0be73b1f9d444997ab7d9ff8ef11a74f" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 32909 } } peers { permanent_uuid: "1d71395342b847c1b2f243a4fd1be929" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 35613 } } }
I20260926 15:08:52.835070 14061 tablet_replica.cc:442] T 7a1a19b0ca2a49c49b5b1626c007d639 P 0be73b1f9d444997ab7d9ff8ef11a74f: This TxnStatusTablet replica's current role is: LEADER
I20260926 15:08:52.845597 14060 tablet_replica.cc:439] T 7a1a19b0ca2a49c49b5b1626c007d639 P a74c4495c62e440bbf55cd02a879911c: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0be73b1f9d444997ab7d9ff8ef11a74f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a74c4495c62e440bbf55cd02a879911c" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 39853 } } peers { permanent_uuid: "0be73b1f9d444997ab7d9ff8ef11a74f" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 32909 } } peers { permanent_uuid: "1d71395342b847c1b2f243a4fd1be929" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 35613 } } }
I20260926 15:08:52.846237 14060 tablet_replica.cc:442] T 7a1a19b0ca2a49c49b5b1626c007d639 P a74c4495c62e440bbf55cd02a879911c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 15:08:52.851132 14062 tablet_replica.cc:439] T 7a1a19b0ca2a49c49b5b1626c007d639 P 1d71395342b847c1b2f243a4fd1be929: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0be73b1f9d444997ab7d9ff8ef11a74f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a74c4495c62e440bbf55cd02a879911c" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 39853 } } peers { permanent_uuid: "0be73b1f9d444997ab7d9ff8ef11a74f" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 32909 } } peers { permanent_uuid: "1d71395342b847c1b2f243a4fd1be929" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 35613 } } }
I20260926 15:08:52.851429 14062 tablet_replica.cc:442] T 7a1a19b0ca2a49c49b5b1626c007d639 P 1d71395342b847c1b2f243a4fd1be929: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 15:08:52.867350 14102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:08:52.858026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.195:35085 (local address 127.12.34.194:32909)
0926 15:08:52.858177 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0926 15:08:52.858182 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:08:52.861649 (+ 3467us) server_negotiation.cc:407] Connection header received
0926 15:08:52.861885 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:08:52.861889 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:08:52.861966 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:08:52.862065 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 15:08:52.862511 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:08:52.863501 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:08:52.865545 (+ 2044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:08:52.866004 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:08:52.866171 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:08:52.866573 (+ 402us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 15:08:52.866671 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 15:08:52.866776 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 15:08:52.866928 (+ 152us) server_negotiation.cc:299] Negotiation successful
0926 15:08:52.867002 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 15:08:52.867434 14114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:08:52.857912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.194:32909 (local address 127.12.34.195:35085)
0926 15:08:52.861529 (+ 3617us) negotiation.cc:107] Waiting for socket to connect
0926 15:08:52.861550 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 15:08:52.861732 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:08:52.862095 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:08:52.862099 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:08:52.862113 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 15:08:52.862354 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:08:52.862363 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:08:52.863884 (+ 1521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:08:52.863891 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:08:52.865360 (+ 1469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:08:52.865374 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:08:52.866183 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:08:52.866189 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:08:52.866338 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:08:52.866396 (+ 58us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 15:08:52.866776 (+ 380us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 15:08:52.866790 (+ 14us) client_negotiation.cc:769] Sending connection context
0926 15:08:52.866885 (+ 95us) client_negotiation.cc:240] Negotiation successful
0926 15:08:52.867045 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3451,"thread_start_us":3351,"threads_started":1}
I20260926 15:08:52.879833 13709 raft_consensus.cc:1261] T 25250331d3e74105b581ff202922a87a P a74c4495c62e440bbf55cd02a879911c [term 1 FOLLOWER]: Refusing update from remote peer 1d71395342b847c1b2f243a4fd1be929: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:08:52.879843 13846 raft_consensus.cc:1261] T 25250331d3e74105b581ff202922a87a P 0be73b1f9d444997ab7d9ff8ef11a74f [term 1 FOLLOWER]: Refusing update from remote peer 1d71395342b847c1b2f243a4fd1be929: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:08:52.880357 14064 consensus_queue.cc:1035] T 25250331d3e74105b581ff202922a87a P 1d71395342b847c1b2f243a4fd1be929 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a74c4495c62e440bbf55cd02a879911c" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 39853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:08:52.880795 14064 consensus_queue.cc:1035] T 25250331d3e74105b581ff202922a87a P 1d71395342b847c1b2f243a4fd1be929 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0be73b1f9d444997ab7d9ff8ef11a74f" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 32909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:08:52.902957 14103 mvcc.cc:205] Tried to move back new op lower bound from 7333623123470823424 to 7333623121976213504. Current Snapshot: MvccSnapshot[applied={T|T < 7333623123470823424}]
I20260926 15:08:52.935297 14094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:08:52.929227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.194:38837 (local address 127.12.34.195:35613)
0926 15:08:52.929383 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0926 15:08:52.929389 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:08:52.929406 (+ 17us) server_negotiation.cc:407] Connection header received
0926 15:08:52.929489 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:08:52.929495 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:08:52.929557 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:08:52.929653 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 15:08:52.930261 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:08:52.930985 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:08:52.933614 (+ 2629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:08:52.934055 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:08:52.934193 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:08:52.934604 (+ 411us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 15:08:52.934706 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 15:08:52.934808 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 15:08:52.934965 (+ 157us) server_negotiation.cc:299] Negotiation successful
0926 15:08:52.935042 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 15:08:52.936754 14118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:08:52.928514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.195:35613 (local address 127.12.34.194:38837)
0926 15:08:52.929161 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0926 15:08:52.929181 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 15:08:52.929312 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:08:52.929675 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:08:52.929679 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:08:52.929690 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 15:08:52.930076 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:08:52.930088 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:08:52.931107 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:08:52.931114 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:08:52.933485 (+ 2371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:08:52.933496 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:08:52.934232 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:08:52.934240 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:08:52.934381 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:08:52.934435 (+ 54us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 15:08:52.934810 (+ 375us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 15:08:52.934823 (+ 13us) client_negotiation.cc:769] Sending connection context
0926 15:08:52.935883 (+ 1060us) client_negotiation.cc:240] Negotiation successful
0926 15:08:52.936031 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":1437,"threads_started":1}
I20260926 15:08:52.970672 13550 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57072:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 15:08:52.970908 13550 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57072:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 15:08:52.973519 13550 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e855e2d27ca44f7fbc65d11d9bc0b942: Sending DeleteTablet for 3 replicas of tablet 25250331d3e74105b581ff202922a87a
I20260926 15:08:52.974937 13826 tablet_service.cc:1514] Processing DeleteTablet for tablet 25250331d3e74105b581ff202922a87a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:08:52 UTC) from {username='kudu', principal='kudu/127.12.34.254@KRBTEST.COM'} at 127.0.0.1:37458
I20260926 15:08:52.975340 13969 tablet_service.cc:1514] Processing DeleteTablet for tablet 25250331d3e74105b581ff202922a87a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:08:52 UTC) from {username='kudu', principal='kudu/127.12.34.254@KRBTEST.COM'} at 127.0.0.1:56728
I20260926 15:08:52.978236 13686 tablet_service.cc:1514] Processing DeleteTablet for tablet 25250331d3e74105b581ff202922a87a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:08:52 UTC) from {username='kudu', principal='kudu/127.12.34.254@KRBTEST.COM'} at 127.0.0.1:46920
I20260926 15:08:52.990681 14121 tablet_replica.cc:330] T 25250331d3e74105b581ff202922a87a P 0be73b1f9d444997ab7d9ff8ef11a74f: stopping tablet replica
I20260926 15:08:52.991074 14121 raft_consensus.cc:2229] T 25250331d3e74105b581ff202922a87a P 0be73b1f9d444997ab7d9ff8ef11a74f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 15:08:52.991448 14121 raft_consensus.cc:2258] T 25250331d3e74105b581ff202922a87a P 0be73b1f9d444997ab7d9ff8ef11a74f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:08:52.992851 14121 ts_tablet_manager.cc:1905] T 25250331d3e74105b581ff202922a87a P 0be73b1f9d444997ab7d9ff8ef11a74f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:08:52.993525 14123 tablet_replica.cc:330] T 25250331d3e74105b581ff202922a87a P a74c4495c62e440bbf55cd02a879911c: stopping tablet replica
I20260926 15:08:52.993708 14123 raft_consensus.cc:2229] T 25250331d3e74105b581ff202922a87a P a74c4495c62e440bbf55cd02a879911c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 15:08:52.993947 14123 raft_consensus.cc:2258] T 25250331d3e74105b581ff202922a87a P a74c4495c62e440bbf55cd02a879911c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:08:52.994997 14123 ts_tablet_manager.cc:1905] T 25250331d3e74105b581ff202922a87a P a74c4495c62e440bbf55cd02a879911c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:08:53.002688 14121 ts_tablet_manager.cc:1918] T 25250331d3e74105b581ff202922a87a P 0be73b1f9d444997ab7d9ff8ef11a74f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 15:08:53.002753 14123 ts_tablet_manager.cc:1918] T 25250331d3e74105b581ff202922a87a P a74c4495c62e440bbf55cd02a879911c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 15:08:53.002877 14123 log.cc:1196] T 25250331d3e74105b581ff202922a87a P a74c4495c62e440bbf55cd02a879911c: Deleting WAL directory at /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435318278681-12427-0/minicluster-data/ts-0/wal/wals/25250331d3e74105b581ff202922a87a
I20260926 15:08:53.002894 14120 tablet_replica.cc:330] T 25250331d3e74105b581ff202922a87a P 1d71395342b847c1b2f243a4fd1be929: stopping tablet replica
I20260926 15:08:53.003032 14121 log.cc:1196] T 25250331d3e74105b581ff202922a87a P 0be73b1f9d444997ab7d9ff8ef11a74f: Deleting WAL directory at /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435318278681-12427-0/minicluster-data/ts-1/wal/wals/25250331d3e74105b581ff202922a87a
I20260926 15:08:53.003077 14120 raft_consensus.cc:2229] T 25250331d3e74105b581ff202922a87a P 1d71395342b847c1b2f243a4fd1be929 [term 1 LEADER]: Raft consensus shutting down.
I20260926 15:08:53.003322 14123 ts_tablet_manager.cc:1939] T 25250331d3e74105b581ff202922a87a P a74c4495c62e440bbf55cd02a879911c: Deleting consensus metadata
I20260926 15:08:53.003491 14120 raft_consensus.cc:2258] T 25250331d3e74105b581ff202922a87a P 1d71395342b847c1b2f243a4fd1be929 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:08:53.003705 14121 ts_tablet_manager.cc:1939] T 25250331d3e74105b581ff202922a87a P 0be73b1f9d444997ab7d9ff8ef11a74f: Deleting consensus metadata
I20260926 15:08:53.004271 13534 catalog_manager.cc:4841] TS a74c4495c62e440bbf55cd02a879911c (127.12.34.193:39853): tablet 25250331d3e74105b581ff202922a87a (table test-table [id=ef314a1a433d49fdbd6bb53718703d76]) successfully deleted
I20260926 15:08:53.005010 14120 ts_tablet_manager.cc:1905] T 25250331d3e74105b581ff202922a87a P 1d71395342b847c1b2f243a4fd1be929: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:08:53.005190 13535 catalog_manager.cc:4841] TS 0be73b1f9d444997ab7d9ff8ef11a74f (127.12.34.194:32909): tablet 25250331d3e74105b581ff202922a87a (table test-table [id=ef314a1a433d49fdbd6bb53718703d76]) successfully deleted
I20260926 15:08:53.011229 14120 ts_tablet_manager.cc:1918] T 25250331d3e74105b581ff202922a87a P 1d71395342b847c1b2f243a4fd1be929: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 15:08:53.011348 14120 log.cc:1196] T 25250331d3e74105b581ff202922a87a P 1d71395342b847c1b2f243a4fd1be929: Deleting WAL directory at /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790435318278681-12427-0/minicluster-data/ts-2/wal/wals/25250331d3e74105b581ff202922a87a
I20260926 15:08:53.011780 14120 ts_tablet_manager.cc:1939] T 25250331d3e74105b581ff202922a87a P 1d71395342b847c1b2f243a4fd1be929: Deleting consensus metadata
I20260926 15:08:53.012763 13536 catalog_manager.cc:4841] TS 1d71395342b847c1b2f243a4fd1be929 (127.12.34.195:35613): tablet 25250331d3e74105b581ff202922a87a (table test-table [id=ef314a1a433d49fdbd6bb53718703d76]) successfully deleted
Sep 26 15:08:53 dist-test-slave-865v krb5kdc[13505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435331, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.12.34.193@KRBTEST.COM
I20260926 15:08:53.021813 14096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:08:53.000552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46924 (local address 127.12.34.193:39853)
0926 15:08:53.000716 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0926 15:08:53.000724 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 15:08:53.004169 (+ 3445us) server_negotiation.cc:407] Connection header received
0926 15:08:53.005570 (+ 1401us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:08:53.005576 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:08:53.005666 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:08:53.005776 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:08:53.007535 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:08:53.008573 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:08:53.010820 (+ 2247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:08:53.011543 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:08:53.011697 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:08:53.016471 (+ 4774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:08:53.016483 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:08:53.016497 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:08:53.016525 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:08:53.019256 (+ 2731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:08:53.019892 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:08:53.019900 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:08:53.019902 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:08:53.020014 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:08:53.020374 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:08:53.020383 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:08:53.020385 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:08:53.020697 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:08:53.021009 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0926 15:08:53.021277 (+ 268us) server_negotiation.cc:299] Negotiation successful
0926 15:08:53.021530 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
W20260926 15:08:53.022531 13752 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46924
I20260926 15:08:53.038445 14089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:08:53.025053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57090 (local address 127.12.34.254:40761)
0926 15:08:53.025286 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0926 15:08:53.025291 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:08:53.025481 (+ 190us) server_negotiation.cc:407] Connection header received
0926 15:08:53.025643 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:08:53.025646 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:08:53.025719 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:08:53.025795 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:08:53.027318 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:08:53.028184 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:08:53.029970 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:08:53.030344 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:08:53.030458 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:08:53.033154 (+ 2696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:08:53.033159 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:08:53.033163 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:08:53.033189 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:08:53.036065 (+ 2876us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:08:53.036766 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:08:53.036769 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:08:53.036771 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:08:53.036869 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:08:53.037253 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:08:53.037257 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:08:53.037258 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:08:53.037501 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:08:53.037655 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0926 15:08:53.038017 (+ 362us) server_negotiation.cc:299] Negotiation successful
0926 15:08:53.038165 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
W20260926 15:08:53.038961 13550 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57090
I20260926 15:08:53.040688 12427 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 13621
I20260926 15:08:53.053251 14096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:08:53.041804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46938 (local address 127.12.34.193:39853)
0926 15:08:53.041971 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0926 15:08:53.041976 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:08:53.041993 (+ 17us) server_negotiation.cc:407] Connection header received
0926 15:08:53.042081 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:08:53.042086 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:08:53.042163 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:08:53.042262 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:08:53.043271 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:08:53.043964 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:08:53.045109 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:08:53.045516 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:08:53.045644 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:08:53.047848 (+ 2204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:08:53.047854 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:08:53.047859 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:08:53.047900 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:08:53.050758 (+ 2858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:08:53.051347 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:08:53.051353 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:08:53.051357 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:08:53.051474 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:08:53.051815 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:08:53.051821 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:08:53.051824 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:08:53.052080 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:08:53.052329 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0926 15:08:53.052643 (+ 314us) server_negotiation.cc:299] Negotiation successful
0926 15:08:53.052855 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20260926 15:08:53.053781 13752 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46938
W20260926 15:08:53.054291 12427 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 13621: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 15:08:53.054497 12427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8uoJGv/build/asan/bin/kudu with pid 13621
I20260926 15:08:53.071892 12427 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 13761
Sep 26 15:08:53 dist-test-slave-865v krb5kdc[13505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435331, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.12.34.194@KRBTEST.COM
I20260926 15:08:53.087286 14102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:08:53.072753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37474 (local address 127.12.34.194:32909)
0926 15:08:53.072911 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0926 15:08:53.072916 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:08:53.072996 (+ 80us) server_negotiation.cc:407] Connection header received
0926 15:08:53.073223 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:08:53.073228 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:08:53.073300 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:08:53.073420 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:08:53.074457 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:08:53.075210 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:08:53.076482 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:08:53.077001 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:08:53.077247 (+ 246us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:08:53.080938 (+ 3691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:08:53.080948 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:08:53.080960 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:08:53.080989 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:08:53.084215 (+ 3226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:08:53.085014 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:08:53.085030 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:08:53.085033 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:08:53.085336 (+ 303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:08:53.085727 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:08:53.085731 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:08:53.085737 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:08:53.086088 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:08:53.086437 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0926 15:08:53.086704 (+ 267us) server_negotiation.cc:299] Negotiation successful
0926 15:08:53.086933 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260926 15:08:53.087954 13891 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37474
W20260926 15:08:53.088632 12427 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 13761: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 15:08:53.088792 12427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8uoJGv/build/asan/bin/kudu with pid 13761
I20260926 15:08:53.107997 12427 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 13900
Sep 26 15:08:53 dist-test-slave-865v krb5kdc[13505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435331, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.12.34.195@KRBTEST.COM
I20260926 15:08:53.123950 14094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:08:53.108623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56740 (local address 127.12.34.195:35613)
0926 15:08:53.108785 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0926 15:08:53.108789 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:08:53.108901 (+ 112us) server_negotiation.cc:407] Connection header received
0926 15:08:53.109069 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:08:53.109073 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:08:53.109180 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:08:53.109262 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:08:53.110651 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:08:53.111510 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:08:53.112869 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:08:53.113363 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:08:53.113457 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:08:53.117955 (+ 4498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:08:53.117969 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:08:53.117986 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:08:53.118017 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:08:53.121465 (+ 3448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:08:53.122178 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:08:53.122184 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:08:53.122187 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:08:53.122319 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:08:53.122728 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:08:53.122732 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:08:53.122734 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:08:53.123099 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:08:53.123317 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0926 15:08:53.123543 (+ 226us) server_negotiation.cc:299] Negotiation successful
0926 15:08:53.123697 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
W20260926 15:08:53.124704 14030 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56740
W20260926 15:08:53.125360 12427 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 13900: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 15:08:53.125506 12427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8uoJGv/build/asan/bin/kudu with pid 13900
I20260926 15:08:53.147181 12427 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 13518
I20260926 15:08:53.148357 13590 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:08:53.262704 12427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8uoJGv/build/asan/bin/kudu with pid 13518
2026-09-26T15:08:53Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4694 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435318278681-12427-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 15:08:53 dist-test-slave-865v krb5kdc[14131](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 15:08:53 dist-test-slave-865v krb5kdc[14131](info): set up 2 sockets
Sep 26 15:08:53 dist-test-slave-865v krb5kdc[14131](info): commencing operation
krb5kdc: starting...
W20260926 15:08:54.345306 12427 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.000s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435318278681-12427-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435318278681-12427-0/krb5kdc/test-user.keytab.
Sep 26 15:08:54 dist-test-slave-865v krb5kdc[14131](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435334, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T15:08:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T15:08:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.34.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.34.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.34.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.254.keytab.
Entry for principal kudu/127.12.34.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.254.keytab.
I20260926 15:08:54.522792 12427 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8uoJGv/build/asan/bin/kudu
/tmp/dist-test-task8uoJGv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435318278681-12427-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435318278681-12427-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435318278681-12427-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.34.254:45313
--webserver_interface=127.12.34.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.254.keytab
--principal=kudu/127.12.34.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.34.212:37255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.34.254:45313
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435318278681-12427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435318278681-12427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435318278681-12427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:08:54.834996 14144 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:08:54.835263 14144 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:08:54.835350 14144 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:08:54.842190 14144 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:08:54.842284 14144 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 15:08:54.842324 14144 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:08:54.842382 14144 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:08:54.842428 14144 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:08:54.842492 14144 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:08:54.851692 14144 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.34.212:37255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435318278681-12427-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435318278681-12427-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.34.254:45313
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.254.keytab
--principal=kudu/127.12.34.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.34.254:45313
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.34.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435318278681-12427-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:08:54.852808 14144 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:08:54.854004 14144 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:08:54.865301 14149 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:08:54.865274 14150 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:08:54.865342 14152 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:08:54.865978 14144 server_base.cc:1034] running on GCE node
I20260926 15:08:54.866756 14144 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:08:54.868126 14144 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:08:54.869323 14144 hybrid_clock.cc:648] HybridClock initialized: now 1790435334869268 us; error 36 us; skew 500 ppm
I20260926 15:08:54.869648 14144 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:08:54 dist-test-slave-865v krb5kdc[14131](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435334, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:08:54.872491 14144 init.cc:377] Logged in from keytab as kudu/127.12.34.254@KRBTEST.COM (short username kudu)
I20260926 15:08:54.873831 14144 webserver.cc:460] Webserver started at http://127.12.34.254:35183/ using document root <none> and password file <none>
I20260926 15:08:54.874449 14144 fs_manager.cc:359] Metadata directory not provided
I20260926 15:08:54.874563 14144 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:08:54.874805 14144 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:08:54.877629 14144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435318278681-12427-0/minicluster-data/master-0/data/instance:
uuid: "737da02a56c94512be48ba241fbb275d"
format_stamp: "Formatted at 2026-09-26 15:08:54 on dist-test-slave-865v"
server_key: "611b594e9bee4137a2ea33b62b5b13e8"
server_key_iv: "22b569548589a69939d288ff3fae0303"
server_key_version: "encryptionkey@0"
I20260926 15:08:54.878293 14144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435318278681-12427-0/minicluster-data/master-0/wal/instance:
uuid: "737da02a56c94512be48ba241fbb275d"
format_stamp: "Formatted at 2026-09-26 15:08:54 on dist-test-slave-865v"
server_key: "611b594e9bee4137a2ea33b62b5b13e8"
server_key_iv: "22b569548589a69939d288ff3fae0303"
server_key_version: "encryptionkey@0"
I20260926 15:08:54.883258 14144 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 15:08:54.886426 14159 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:08:54.887639 14144 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 15:08:54.887809 14144 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435318278681-12427-0/minicluster-data/master-0/data,/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435318278681-12427-0/minicluster-data/master-0/wal
uuid: "737da02a56c94512be48ba241fbb275d"
format_stamp: "Formatted at 2026-09-26 15:08:54 on dist-test-slave-865v"
server_key: "611b594e9bee4137a2ea33b62b5b13e8"
server_key_iv: "22b569548589a69939d288ff3fae0303"
server_key_version: "encryptionkey@0"
I20260926 15:08:54.888018 14144 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435318278681-12427-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435318278681-12427-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435318278681-12427-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:08:54.904312 14144 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:08:54.908608 14144 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:08:54.908900 14144 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:08:54.931005 14144 rpc_server.cc:307] RPC server started. Bound to: 127.12.34.254:45313
I20260926 15:08:54.931023 14220 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.34.254:45313 every 8 connection(s)
I20260926 15:08:54.932327 14144 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
I20260926 15:08:54.935870 14221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:08:54.938509 12427 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8uoJGv/build/asan/bin/kudu as pid 14144
I20260926 15:08:54.938692 12427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435318278681-12427-0/minicluster-data/master-0/wal/instance
I20260926 15:08:54.939460 12427 external_mini_cluster.cc:1428] Setting key 4b317364b1c46b1d88c0199c017139c2
I20260926 15:08:54.947657 14221 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 737da02a56c94512be48ba241fbb275d: Bootstrap starting.
I20260926 15:08:54.950632 14221 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 737da02a56c94512be48ba241fbb275d: Neither blocks nor log segments found. Creating new log.
I20260926 15:08:54.951581 14221 log.cc:824] T 00000000000000000000000000000000 P 737da02a56c94512be48ba241fbb275d: Log is configured to *not* fsync() on all Append() calls
Sep 26 15:08:54 dist-test-slave-865v krb5kdc[14131](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435334, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.34.254@KRBTEST.COM
I20260926 15:08:54.954249 14221 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 737da02a56c94512be48ba241fbb275d: No bootstrap required, opened a new log
I20260926 15:08:54.961479 14224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:08:54.941494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46514 (local address 127.12.34.254:45313)
0926 15:08:54.942234 (+ 740us) server_negotiation.cc:206] Beginning negotiation
0926 15:08:54.942246 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 15:08:54.942283 (+ 37us) server_negotiation.cc:407] Connection header received
0926 15:08:54.943873 (+ 1590us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:08:54.943893 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:08:54.944240 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:08:54.944554 (+ 314us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:08:54.946242 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:08:54.947683 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:08:54.949073 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:08:54.949791 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:08:54.950004 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:08:54.953908 (+ 3904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:08:54.953915 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:08:54.953928 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:08:54.953962 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:08:54.957231 (+ 3269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:08:54.958027 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:08:54.958041 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:08:54.958049 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:08:54.958221 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:08:54.958638 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:08:54.958643 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:08:54.958655 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:08:54.959175 (+ 520us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:08:54.959528 (+ 353us) server_negotiation.cc:1035] Waiting for connection context
0926 15:08:54.959930 (+ 402us) server_negotiation.cc:299] Negotiation successful
0926 15:08:54.960376 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":554,"thread_start_us":364,"threads_started":1}
I20260926 15:08:54.962093 14221 raft_consensus.cc:348] T 00000000000000000000000000000000 P 737da02a56c94512be48ba241fbb275d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "737da02a56c94512be48ba241fbb275d" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 45313 } }
I20260926 15:08:54.962430 14221 raft_consensus.cc:374] T 00000000000000000000000000000000 P 737da02a56c94512be48ba241fbb275d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:08:54.962541 14221 raft_consensus.cc:729] T 00000000000000000000000000000000 P 737da02a56c94512be48ba241fbb275d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 737da02a56c94512be48ba241fbb275d, State: Initialized, Role: FOLLOWER
I20260926 15:08:54.963088 14221 consensus_queue.cc:260] T 00000000000000000000000000000000 P 737da02a56c94512be48ba241fbb275d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "737da02a56c94512be48ba241fbb275d" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 45313 } }
I20260926 15:08:54.963311 14221 raft_consensus.cc:388] T 00000000000000000000000000000000 P 737da02a56c94512be48ba241fbb275d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 15:08:54.963459 14221 raft_consensus.cc:482] T 00000000000000000000000000000000 P 737da02a56c94512be48ba241fbb275d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 15:08:54.963629 14221 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 737da02a56c94512be48ba241fbb275d [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:08:54.965875 14221 raft_consensus.cc:504] T 00000000000000000000000000000000 P 737da02a56c94512be48ba241fbb275d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "737da02a56c94512be48ba241fbb275d" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 45313 } }
I20260926 15:08:54.966326 14221 leader_election.cc:302] T 00000000000000000000000000000000 P 737da02a56c94512be48ba241fbb275d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 737da02a56c94512be48ba241fbb275d; no voters:
I20260926 15:08:54.966869 14221 leader_election.cc:288] T 00000000000000000000000000000000 P 737da02a56c94512be48ba241fbb275d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 15:08:54.966996 14226 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 737da02a56c94512be48ba241fbb275d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:08:54.968221 14226 raft_consensus.cc:686] T 00000000000000000000000000000000 P 737da02a56c94512be48ba241fbb275d [term 1 LEADER]: Becoming Leader. State: Replica: 737da02a56c94512be48ba241fbb275d, State: Running, Role: LEADER
I20260926 15:08:54.968765 14221 sys_catalog.cc:564] T 00000000000000000000000000000000 P 737da02a56c94512be48ba241fbb275d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:08:54.968870 14226 consensus_queue.cc:237] T 00000000000000000000000000000000 P 737da02a56c94512be48ba241fbb275d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "737da02a56c94512be48ba241fbb275d" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 45313 } }
I20260926 15:08:54.971438 14228 sys_catalog.cc:455] T 00000000000000000000000000000000 P 737da02a56c94512be48ba241fbb275d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 737da02a56c94512be48ba241fbb275d. Latest consensus state: current_term: 1 leader_uuid: "737da02a56c94512be48ba241fbb275d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "737da02a56c94512be48ba241fbb275d" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 45313 } } }
I20260926 15:08:54.971760 14228 sys_catalog.cc:458] T 00000000000000000000000000000000 P 737da02a56c94512be48ba241fbb275d [sys.catalog]: This master's current role is: LEADER
I20260926 15:08:54.972355 14227 sys_catalog.cc:455] T 00000000000000000000000000000000 P 737da02a56c94512be48ba241fbb275d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "737da02a56c94512be48ba241fbb275d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "737da02a56c94512be48ba241fbb275d" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 45313 } } }
I20260926 15:08:54.972608 14227 sys_catalog.cc:458] T 00000000000000000000000000000000 P 737da02a56c94512be48ba241fbb275d [sys.catalog]: This master's current role is: LEADER
I20260926 15:08:54.977737 14236 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:08:54.981990 14236 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 15:08:54.991827 14236 catalog_manager.cc:1347] Generated new cluster ID: 6261fe9ac5ec4bd881802456cac1eca0
I20260926 15:08:54.991935 14236 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 15:08:55.029728 14236 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 15:08:55.031203 14236 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:08:55.037752 14236 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 737da02a56c94512be48ba241fbb275d: Generated new TSK 0
I20260926 15:08:55.038635 14236 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.34.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.34.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.34.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.193.keytab.
Entry for principal kudu/127.12.34.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.193.keytab.
I20260926 15:08:55.097375 12427 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8uoJGv/build/asan/bin/kudu
/tmp/dist-test-task8uoJGv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435318278681-12427-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435318278681-12427-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435318278681-12427-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435318278681-12427-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.34.193:0
--local_ip_for_outbound_sockets=127.12.34.193
--webserver_interface=127.12.34.193
--webserver_port=0
--tserver_master_addrs=127.12.34.254:45313
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.193.keytab
--principal=kudu/127.12.34.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.34.212:37255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435318278681-12427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435318278681-12427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435318278681-12427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:08:55.408879 14247 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:08:55.409196 14247 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:08:55.409353 14247 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:08:55.416368 14247 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:08:55.416458 14247 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:08:55.416615 14247 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 15:08:55.416690 14247 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 15:08:55.416806 14247 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.34.193
I20260926 15:08:55.426178 14247 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.34.212:37255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435318278681-12427-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435318278681-12427-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.193.keytab
--principal=kudu/127.12.34.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.34.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435318278681-12427-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.34.193
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.12.34.254:45313
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.34.193
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435318278681-12427-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:08:55.427260 14247 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:08:55.428438 14247 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:08:55.440816 14255 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:08:55.440908 14252 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:08:55.440917 14253 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:08:55.441471 14247 server_base.cc:1034] running on GCE node
I20260926 15:08:55.442021 14247 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:08:55.442850 14247 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:08:55.444067 14247 hybrid_clock.cc:648] HybridClock initialized: now 1790435335444027 us; error 50 us; skew 500 ppm
I20260926 15:08:55.444360 14247 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:08:55 dist-test-slave-865v krb5kdc[14131](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435335, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:08:55.447283 14247 init.cc:377] Logged in from keytab as kudu/127.12.34.193@KRBTEST.COM (short username kudu)
I20260926 15:08:55.448596 14247 webserver.cc:460] Webserver started at http://127.12.34.193:45179/ using document root <none> and password file <none>
I20260926 15:08:55.449285 14247 fs_manager.cc:359] Metadata directory not provided
I20260926 15:08:55.449414 14247 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:08:55.449704 14247 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:08:55.452314 14247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435318278681-12427-0/minicluster-data/ts-0/data/instance:
uuid: "26a4aa858ba14954b2e505694f779a86"
format_stamp: "Formatted at 2026-09-26 15:08:55 on dist-test-slave-865v"
server_key: "30645690bedb0a59409c2c608542627c"
server_key_iv: "7bc896705788fe8ac7d6b3b1ba4c907d"
server_key_version: "encryptionkey@0"
I20260926 15:08:55.453023 14247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435318278681-12427-0/minicluster-data/ts-0/wal/instance:
uuid: "26a4aa858ba14954b2e505694f779a86"
format_stamp: "Formatted at 2026-09-26 15:08:55 on dist-test-slave-865v"
server_key: "30645690bedb0a59409c2c608542627c"
server_key_iv: "7bc896705788fe8ac7d6b3b1ba4c907d"
server_key_version: "encryptionkey@0"
I20260926 15:08:55.457691 14247 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260926 15:08:55.460961 14262 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:08:55.462195 14247 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260926 15:08:55.462368 14247 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435318278681-12427-0/minicluster-data/ts-0/data,/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435318278681-12427-0/minicluster-data/ts-0/wal
uuid: "26a4aa858ba14954b2e505694f779a86"
format_stamp: "Formatted at 2026-09-26 15:08:55 on dist-test-slave-865v"
server_key: "30645690bedb0a59409c2c608542627c"
server_key_iv: "7bc896705788fe8ac7d6b3b1ba4c907d"
server_key_version: "encryptionkey@0"
I20260926 15:08:55.462601 14247 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435318278681-12427-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435318278681-12427-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435318278681-12427-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:08:55.493877 14247 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:08:55.497709 14247 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:08:55.497980 14247 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:08:55.500622 14247 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:08:55.500757 14247 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:08:55.500876 14247 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:08:55.500972 14247 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 15:08:55 dist-test-slave-865v krb5kdc[14131](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435335, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.193@KRBTEST.COM for kudu/127.12.34.254@KRBTEST.COM
I20260926 15:08:55.525034 14271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:08:55.501739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.193:44347 (local address 127.12.34.254:45313)
0926 15:08:55.502181 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0926 15:08:55.502188 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:08:55.503355 (+ 1167us) server_negotiation.cc:407] Connection header received
0926 15:08:55.506573 (+ 3218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:08:55.506582 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:08:55.506673 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:08:55.506809 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:08:55.508982 (+ 2173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:08:55.510070 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:08:55.512010 (+ 1940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:08:55.512496 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:08:55.512633 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:08:55.517879 (+ 5246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:08:55.517885 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:08:55.517890 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:08:55.517931 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:08:55.521444 (+ 3513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:08:55.522315 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:08:55.522322 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:08:55.522326 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:08:55.522448 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:08:55.522992 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:08:55.522997 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:08:55.523000 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:08:55.523400 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:08:55.523549 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0926 15:08:55.524475 (+ 926us) server_negotiation.cc:299] Negotiation successful
0926 15:08:55.524675 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":219,"threads_started":1}
I20260926 15:08:55.526006 14272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:08:55.502335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:45313 (local address 127.12.34.193:44347)
0926 15:08:55.503165 (+ 830us) negotiation.cc:107] Waiting for socket to connect
0926 15:08:55.503226 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0926 15:08:55.506166 (+ 2940us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:08:55.507058 (+ 892us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:08:55.507066 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:08:55.507908 (+ 842us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:08:55.508807 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:08:55.508833 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:08:55.510215 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:08:55.510221 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:08:55.511861 (+ 1640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:08:55.511874 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:08:55.512784 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:08:55.512794 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:08:55.513186 (+ 392us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:08:55.513983 (+ 797us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:08:55.514022 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:08:55.517627 (+ 3605us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:08:55.521561 (+ 3934us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:08:55.521569 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:08:55.521588 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:08:55.522189 (+ 601us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:08:55.522551 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:08:55.522558 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:08:55.522562 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:08:55.522871 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:08:55.523614 (+ 743us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:08:55.523623 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:08:55.524113 (+ 490us) client_negotiation.cc:769] Sending connection context
0926 15:08:55.524400 (+ 287us) client_negotiation.cc:240] Negotiation successful
0926 15:08:55.524884 (+ 484us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":322,"threads_started":1}
I20260926 15:08:55.544515 14271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:08:55.532157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.193:58249 (local address 127.12.34.254:45313)
0926 15:08:55.532307 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0926 15:08:55.532310 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 15:08:55.532956 (+ 646us) server_negotiation.cc:407] Connection header received
0926 15:08:55.533190 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:08:55.533196 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:08:55.533304 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:08:55.533397 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:08:55.534256 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:08:55.534943 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:08:55.536185 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:08:55.536583 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:08:55.536748 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:08:55.538903 (+ 2155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:08:55.538909 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:08:55.538914 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:08:55.538951 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:08:55.542014 (+ 3063us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:08:55.542621 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:08:55.542628 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:08:55.542632 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:08:55.542755 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:08:55.543083 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:08:55.543090 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:08:55.543093 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:08:55.543330 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:08:55.543589 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0926 15:08:55.543964 (+ 375us) server_negotiation.cc:299] Negotiation successful
0926 15:08:55.544179 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 15:08:55.544616 14349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:08:55.532281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:45313 (local address 127.12.34.193:58249)
0926 15:08:55.532846 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0926 15:08:55.532865 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 15:08:55.533034 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:08:55.533423 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:08:55.533427 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:08:55.533849 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:08:55.534125 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:08:55.534133 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:08:55.535067 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:08:55.535073 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:08:55.536055 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:08:55.536065 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:08:55.536815 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:08:55.536838 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:08:55.537014 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:08:55.537705 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:08:55.537733 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:08:55.538736 (+ 1003us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:08:55.542136 (+ 3400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:08:55.542141 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:08:55.542145 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:08:55.542512 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:08:55.542855 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:08:55.542860 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:08:55.542864 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:08:55.542985 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:08:55.543554 (+ 569us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:08:55.543559 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:08:55.543728 (+ 169us) client_negotiation.cc:769] Sending connection context
0926 15:08:55.543864 (+ 136us) client_negotiation.cc:240] Negotiation successful
0926 15:08:55.544179 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":313,"threads_started":1}
I20260926 15:08:55.551069 14247 rpc_server.cc:307] RPC server started. Bound to: 127.12.34.193:45693
I20260926 15:08:55.551097 14382 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.34.193:45693 every 8 connection(s)
I20260926 15:08:55.552181 14247 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435318278681-12427-0/minicluster-data/ts-0/data/info.pb
I20260926 15:08:55.554044 12427 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8uoJGv/build/asan/bin/kudu as pid 14247
I20260926 15:08:55.554244 12427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435318278681-12427-0/minicluster-data/ts-0/wal/instance
I20260926 15:08:55.554286 14383 heartbeater.cc:347] Connected to a master server at 127.12.34.254:45313
I20260926 15:08:55.554536 14383 heartbeater.cc:464] Registering TS with master...
I20260926 15:08:55.555027 12427 external_mini_cluster.cc:1428] Setting key 1a4e7cba94f120736ab6064aaf684856
I20260926 15:08:55.555218 14383 heartbeater.cc:511] Master 127.12.34.254:45313 requested a full tablet report, sending...
I20260926 15:08:55.557144 14175 ts_manager.cc:194] Registered new tserver with Master: 26a4aa858ba14954b2e505694f779a86 (127.12.34.193:45693)
I20260926 15:08:55.558512 14175 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.34.193@KRBTEST.COM'} at 127.12.34.193:44347
WARNING: no policy specified for kudu/127.12.34.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.34.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.34.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.194.keytab.
Entry for principal kudu/127.12.34.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.194.keytab.
I20260926 15:08:55.596410 12427 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8uoJGv/build/asan/bin/kudu
/tmp/dist-test-task8uoJGv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435318278681-12427-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435318278681-12427-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435318278681-12427-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435318278681-12427-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.34.194:0
--local_ip_for_outbound_sockets=127.12.34.194
--webserver_interface=127.12.34.194
--webserver_port=0
--tserver_master_addrs=127.12.34.254:45313
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.194.keytab
--principal=kudu/127.12.34.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.34.212:37255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435318278681-12427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435318278681-12427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435318278681-12427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:08:55.910290 14387 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:08:55.910633 14387 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:08:55.910737 14387 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:08:55.917872 14387 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:08:55.917979 14387 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:08:55.918062 14387 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 15:08:55.918114 14387 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 15:08:55.918175 14387 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.34.194
I20260926 15:08:55.927609 14387 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.34.212:37255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435318278681-12427-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435318278681-12427-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.194.keytab
--principal=kudu/127.12.34.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.34.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435318278681-12427-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.34.194
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.12.34.254:45313
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.34.194
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435318278681-12427-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:08:55.928818 14387 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:08:55.930033 14387 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:08:55.943217 14392 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:08:55.943311 14393 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:08:55.944742 14395 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:08:55.947290 14387 server_base.cc:1034] running on GCE node
I20260926 15:08:55.947821 14387 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:08:55.948884 14387 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:08:55.950162 14387 hybrid_clock.cc:648] HybridClock initialized: now 1790435335950120 us; error 58 us; skew 500 ppm
I20260926 15:08:55.950480 14387 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:08:55 dist-test-slave-865v krb5kdc[14131](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435335, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:08:55.953616 14387 init.cc:377] Logged in from keytab as kudu/127.12.34.194@KRBTEST.COM (short username kudu)
I20260926 15:08:55.954900 14387 webserver.cc:460] Webserver started at http://127.12.34.194:45661/ using document root <none> and password file <none>
I20260926 15:08:55.955510 14387 fs_manager.cc:359] Metadata directory not provided
I20260926 15:08:55.955601 14387 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:08:55.955863 14387 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:08:55.958527 14387 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435318278681-12427-0/minicluster-data/ts-1/data/instance:
uuid: "09ca737613794860ac9e00105c0b5e73"
format_stamp: "Formatted at 2026-09-26 15:08:55 on dist-test-slave-865v"
server_key: "d74e5df4280ace4cf91fd9f135274ca9"
server_key_iv: "f4353395554680919b4f594794171aa9"
server_key_version: "encryptionkey@0"
I20260926 15:08:55.959177 14387 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435318278681-12427-0/minicluster-data/ts-1/wal/instance:
uuid: "09ca737613794860ac9e00105c0b5e73"
format_stamp: "Formatted at 2026-09-26 15:08:55 on dist-test-slave-865v"
server_key: "d74e5df4280ace4cf91fd9f135274ca9"
server_key_iv: "f4353395554680919b4f594794171aa9"
server_key_version: "encryptionkey@0"
I20260926 15:08:55.963883 14387 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 15:08:55.967178 14402 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:08:55.968428 14387 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 15:08:55.968582 14387 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435318278681-12427-0/minicluster-data/ts-1/data,/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435318278681-12427-0/minicluster-data/ts-1/wal
uuid: "09ca737613794860ac9e00105c0b5e73"
format_stamp: "Formatted at 2026-09-26 15:08:55 on dist-test-slave-865v"
server_key: "d74e5df4280ace4cf91fd9f135274ca9"
server_key_iv: "f4353395554680919b4f594794171aa9"
server_key_version: "encryptionkey@0"
I20260926 15:08:55.968729 14387 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435318278681-12427-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435318278681-12427-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435318278681-12427-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:08:55.993920 14387 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:08:55.998014 14387 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:08:55.998256 14387 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:08:56.000574 14387 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:08:56.000721 14387 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:08:56.000851 14387 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:08:56.000939 14387 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 15:08:56 dist-test-slave-865v krb5kdc[14131](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435335, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.194@KRBTEST.COM for kudu/127.12.34.254@KRBTEST.COM
I20260926 15:08:56.022684 14271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:08:56.001419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.194:57693 (local address 127.12.34.254:45313)
0926 15:08:56.001617 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0926 15:08:56.001622 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:08:56.003263 (+ 1641us) server_negotiation.cc:407] Connection header received
0926 15:08:56.006440 (+ 3177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:08:56.006447 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:08:56.006555 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:08:56.006671 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:08:56.008519 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:08:56.009306 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:08:56.012041 (+ 2735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:08:56.012469 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:08:56.012605 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:08:56.017346 (+ 4741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:08:56.017351 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:08:56.017355 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:08:56.017386 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:08:56.019870 (+ 2484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:08:56.020516 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:08:56.020524 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:08:56.020529 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:08:56.020643 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:08:56.021038 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:08:56.021045 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:08:56.021051 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:08:56.021344 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:08:56.021496 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0926 15:08:56.022124 (+ 628us) server_negotiation.cc:299] Negotiation successful
0926 15:08:56.022329 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 15:08:56.023626 14411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:08:56.002239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:45313 (local address 127.12.34.194:57693)
0926 15:08:56.003087 (+ 848us) negotiation.cc:107] Waiting for socket to connect
0926 15:08:56.003132 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 15:08:56.006096 (+ 2964us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:08:56.006963 (+ 867us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:08:56.006977 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:08:56.007795 (+ 818us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:08:56.008359 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:08:56.008376 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:08:56.009515 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:08:56.009520 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:08:56.011889 (+ 2369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:08:56.011907 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:08:56.012708 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:08:56.012716 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:08:56.013225 (+ 509us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:08:56.013997 (+ 772us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:08:56.014019 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:08:56.017120 (+ 3101us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:08:56.020007 (+ 2887us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:08:56.020015 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:08:56.020029 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:08:56.020402 (+ 373us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:08:56.020743 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:08:56.020749 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:08:56.020753 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:08:56.020929 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:08:56.021465 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:08:56.021472 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:08:56.021793 (+ 321us) client_negotiation.cc:769] Sending connection context
0926 15:08:56.022055 (+ 262us) client_negotiation.cc:240] Negotiation successful
0926 15:08:56.022515 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":329,"threads_started":1}
I20260926 15:08:56.049054 14271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:08:56.029710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.194:45697 (local address 127.12.34.254:45313)
0926 15:08:56.029880 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0926 15:08:56.029887 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:08:56.033674 (+ 3787us) server_negotiation.cc:407] Connection header received
0926 15:08:56.033877 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:08:56.033882 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:08:56.033944 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:08:56.034037 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:08:56.035331 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:08:56.036244 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:08:56.038715 (+ 2471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:08:56.039278 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:08:56.039380 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:08:56.041931 (+ 2551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:08:56.041941 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:08:56.041948 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:08:56.041996 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:08:56.045858 (+ 3862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:08:56.046593 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:08:56.046602 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:08:56.046608 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:08:56.046770 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:08:56.047231 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:08:56.047236 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:08:56.047238 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:08:56.047638 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:08:56.047789 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0926 15:08:56.048490 (+ 701us) server_negotiation.cc:299] Negotiation successful
0926 15:08:56.048705 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 15:08:56.049208 14459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:08:56.029833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:45313 (local address 127.12.34.194:45697)
0926 15:08:56.033579 (+ 3746us) negotiation.cc:107] Waiting for socket to connect
0926 15:08:56.033597 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 15:08:56.033744 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:08:56.034056 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:08:56.034060 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:08:56.034722 (+ 662us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:08:56.035124 (+ 402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:08:56.035133 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:08:56.036768 (+ 1635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:08:56.036771 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:08:56.038523 (+ 1752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:08:56.038545 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:08:56.039662 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:08:56.039673 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:08:56.039905 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:08:56.040675 (+ 770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:08:56.040699 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:08:56.041729 (+ 1030us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:08:56.046011 (+ 4282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:08:56.046015 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:08:56.046017 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:08:56.046443 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:08:56.046911 (+ 468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:08:56.046917 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:08:56.046923 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:08:56.047106 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:08:56.047844 (+ 738us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:08:56.047848 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:08:56.048112 (+ 264us) client_negotiation.cc:769] Sending connection context
0926 15:08:56.048357 (+ 245us) client_negotiation.cc:240] Negotiation successful
0926 15:08:56.048718 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3594,"thread_start_us":3408,"threads_started":1}
I20260926 15:08:56.057188 14387 rpc_server.cc:307] RPC server started. Bound to: 127.12.34.194:41037
I20260926 15:08:56.057267 14521 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.34.194:41037 every 8 connection(s)
I20260926 15:08:56.058336 14387 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435318278681-12427-0/minicluster-data/ts-1/data/info.pb
I20260926 15:08:56.060199 14522 heartbeater.cc:347] Connected to a master server at 127.12.34.254:45313
I20260926 15:08:56.060418 14522 heartbeater.cc:464] Registering TS with master...
I20260926 15:08:56.061007 14522 heartbeater.cc:511] Master 127.12.34.254:45313 requested a full tablet report, sending...
I20260926 15:08:56.062253 14175 ts_manager.cc:194] Registered new tserver with Master: 09ca737613794860ac9e00105c0b5e73 (127.12.34.194:41037)
I20260926 15:08:56.062865 12427 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8uoJGv/build/asan/bin/kudu as pid 14387
I20260926 15:08:56.063050 12427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435318278681-12427-0/minicluster-data/ts-1/wal/instance
I20260926 15:08:56.063125 14175 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.34.194@KRBTEST.COM'} at 127.12.34.194:57693
I20260926 15:08:56.063519 12427 external_mini_cluster.cc:1428] Setting key fd6477de0220e466d335f3db1f0d6683
WARNING: no policy specified for kudu/127.12.34.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.34.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.34.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.195.keytab.
Entry for principal kudu/127.12.34.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.195.keytab.
I20260926 15:08:56.102553 12427 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8uoJGv/build/asan/bin/kudu
/tmp/dist-test-task8uoJGv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435318278681-12427-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435318278681-12427-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435318278681-12427-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435318278681-12427-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.34.195:0
--local_ip_for_outbound_sockets=127.12.34.195
--webserver_interface=127.12.34.195
--webserver_port=0
--tserver_master_addrs=127.12.34.254:45313
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.195.keytab
--principal=kudu/127.12.34.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.34.212:37255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435318278681-12427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435318278681-12427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435318278681-12427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:08:56.413337 14526 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:08:56.413605 14526 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:08:56.413753 14526 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:08:56.420715 14526 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:08:56.420814 14526 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:08:56.420951 14526 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 15:08:56.421011 14526 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 15:08:56.421087 14526 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.34.195
I20260926 15:08:56.430699 14526 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.34.212:37255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435318278681-12427-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435318278681-12427-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.195.keytab
--principal=kudu/127.12.34.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.34.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435318278681-12427-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.34.195
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.12.34.254:45313
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.34.195
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435318278681-12427-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:08:56.431890 14526 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:08:56.433151 14526 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:08:56.445473 14531 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:08:56.445432 14534 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:08:56.445538 14532 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:08:56.446398 14526 server_base.cc:1034] running on GCE node
I20260926 15:08:56.447088 14526 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:08:56.447978 14526 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:08:56.449168 14526 hybrid_clock.cc:648] HybridClock initialized: now 1790435336449121 us; error 38 us; skew 500 ppm
I20260926 15:08:56.449456 14526 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:08:56 dist-test-slave-865v krb5kdc[14131](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435336, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:08:56.458247 14526 init.cc:377] Logged in from keytab as kudu/127.12.34.195@KRBTEST.COM (short username kudu)
I20260926 15:08:56.459731 14526 webserver.cc:460] Webserver started at http://127.12.34.195:45593/ using document root <none> and password file <none>
I20260926 15:08:56.460388 14526 fs_manager.cc:359] Metadata directory not provided
I20260926 15:08:56.460523 14526 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:08:56.460809 14526 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:08:56.463755 14526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435318278681-12427-0/minicluster-data/ts-2/data/instance:
uuid: "8b4bb0b1671444b7b6c917963e957b6d"
format_stamp: "Formatted at 2026-09-26 15:08:56 on dist-test-slave-865v"
server_key: "cfa743602553346ceb2c2b3c31ff1e3c"
server_key_iv: "5f6dfb513311ca6c96b790d3db211a46"
server_key_version: "encryptionkey@0"
I20260926 15:08:56.464504 14526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435318278681-12427-0/minicluster-data/ts-2/wal/instance:
uuid: "8b4bb0b1671444b7b6c917963e957b6d"
format_stamp: "Formatted at 2026-09-26 15:08:56 on dist-test-slave-865v"
server_key: "cfa743602553346ceb2c2b3c31ff1e3c"
server_key_iv: "5f6dfb513311ca6c96b790d3db211a46"
server_key_version: "encryptionkey@0"
I20260926 15:08:56.469583 14526 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260926 15:08:56.473177 14541 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:08:56.474434 14526 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 15:08:56.474637 14526 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435318278681-12427-0/minicluster-data/ts-2/data,/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435318278681-12427-0/minicluster-data/ts-2/wal
uuid: "8b4bb0b1671444b7b6c917963e957b6d"
format_stamp: "Formatted at 2026-09-26 15:08:56 on dist-test-slave-865v"
server_key: "cfa743602553346ceb2c2b3c31ff1e3c"
server_key_iv: "5f6dfb513311ca6c96b790d3db211a46"
server_key_version: "encryptionkey@0"
I20260926 15:08:56.474835 14526 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435318278681-12427-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435318278681-12427-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435318278681-12427-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:08:56.502950 14526 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:08:56.507009 14526 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:08:56.507287 14526 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:08:56.510044 14526 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:08:56.510182 14526 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:08:56.510313 14526 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:08:56.510432 14526 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
Sep 26 15:08:56 dist-test-slave-865v krb5kdc[14131](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435336, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.195@KRBTEST.COM for kudu/127.12.34.254@KRBTEST.COM
I20260926 15:08:56.531097 14271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:08:56.510539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.195:48735 (local address 127.12.34.254:45313)
0926 15:08:56.510714 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0926 15:08:56.510723 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 15:08:56.512061 (+ 1338us) server_negotiation.cc:407] Connection header received
0926 15:08:56.513953 (+ 1892us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:08:56.513959 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:08:56.514052 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:08:56.514153 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:08:56.516194 (+ 2041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:08:56.517169 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:08:56.520290 (+ 3121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:08:56.520796 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:08:56.520948 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:08:56.524894 (+ 3946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:08:56.524898 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:08:56.524901 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:08:56.524931 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:08:56.527886 (+ 2955us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:08:56.528625 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:08:56.528630 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:08:56.528634 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:08:56.528750 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:08:56.529181 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:08:56.529189 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:08:56.529192 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:08:56.529469 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:08:56.529637 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0926 15:08:56.530689 (+ 1052us) server_negotiation.cc:299] Negotiation successful
0926 15:08:56.530845 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 15:08:56.533061 14550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:08:56.511130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:45313 (local address 127.12.34.195:48735)
0926 15:08:56.511902 (+ 772us) negotiation.cc:107] Waiting for socket to connect
0926 15:08:56.511941 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 15:08:56.513700 (+ 1759us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:08:56.514402 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:08:56.514416 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:08:56.515190 (+ 774us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:08:56.516030 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:08:56.516048 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:08:56.517359 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:08:56.517371 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:08:56.520119 (+ 2748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:08:56.520139 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:08:56.521065 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:08:56.521095 (+ 30us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:08:56.521468 (+ 373us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:08:56.522295 (+ 827us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:08:56.522340 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:08:56.524668 (+ 2328us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:08:56.528021 (+ 3353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:08:56.528036 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:08:56.528050 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:08:56.528510 (+ 460us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:08:56.528867 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:08:56.528872 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:08:56.528877 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:08:56.529050 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:08:56.529637 (+ 587us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:08:56.529645 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:08:56.530189 (+ 544us) client_negotiation.cc:769] Sending connection context
0926 15:08:56.530566 (+ 377us) client_negotiation.cc:240] Negotiation successful
0926 15:08:56.531238 (+ 672us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":342,"threads_started":1}
I20260926 15:08:56.554209 14608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:08:56.539257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:45313 (local address 127.12.34.195:45797)
0926 15:08:56.539885 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0926 15:08:56.539902 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 15:08:56.540017 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:08:56.540407 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:08:56.540410 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:08:56.540825 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:08:56.541146 (+ 321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:08:56.541159 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:08:56.542383 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:08:56.542387 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:08:56.543887 (+ 1500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:08:56.543898 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:08:56.544625 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:08:56.544633 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:08:56.544789 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:08:56.545465 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:08:56.545498 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:08:56.546562 (+ 1064us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:08:56.551224 (+ 4662us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:08:56.551229 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:08:56.551232 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:08:56.551804 (+ 572us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:08:56.552263 (+ 459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:08:56.552267 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:08:56.552270 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:08:56.552405 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:08:56.553122 (+ 717us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:08:56.553128 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:08:56.553313 (+ 185us) client_negotiation.cc:769] Sending connection context
0926 15:08:56.553466 (+ 153us) client_negotiation.cc:240] Negotiation successful
0926 15:08:56.553784 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":355,"threads_started":1}
I20260926 15:08:56.554278 14271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:08:56.539053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.195:45797 (local address 127.12.34.254:45313)
0926 15:08:56.539193 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0926 15:08:56.539197 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:08:56.539978 (+ 781us) server_negotiation.cc:407] Connection header received
0926 15:08:56.540136 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:08:56.540144 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:08:56.540240 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:08:56.540364 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:08:56.541276 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:08:56.542206 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:08:56.544043 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:08:56.544444 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:08:56.544570 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:08:56.546833 (+ 2263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:08:56.546842 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:08:56.546848 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:08:56.546895 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:08:56.551072 (+ 4177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:08:56.551932 (+ 860us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:08:56.551940 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:08:56.551946 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:08:56.552129 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:08:56.552520 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:08:56.552528 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:08:56.552533 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:08:56.552961 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:08:56.553174 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0926 15:08:56.553588 (+ 414us) server_negotiation.cc:299] Negotiation successful
0926 15:08:56.553802 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 15:08:56.560825 14383 heartbeater.cc:503] Master 127.12.34.254:45313 was elected leader, sending a full tablet report...
I20260926 15:08:56.579679 14526 rpc_server.cc:307] RPC server started. Bound to: 127.12.34.195:39017
I20260926 15:08:56.579694 14660 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.34.195:39017 every 8 connection(s)
I20260926 15:08:56.581127 14526 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435318278681-12427-0/minicluster-data/ts-2/data/info.pb
I20260926 15:08:56.583104 14661 heartbeater.cc:347] Connected to a master server at 127.12.34.254:45313
I20260926 15:08:56.583335 14661 heartbeater.cc:464] Registering TS with master...
I20260926 15:08:56.583937 14661 heartbeater.cc:511] Master 127.12.34.254:45313 requested a full tablet report, sending...
I20260926 15:08:56.585315 14175 ts_manager.cc:194] Registered new tserver with Master: 8b4bb0b1671444b7b6c917963e957b6d (127.12.34.195:39017)
I20260926 15:08:56.586578 14175 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.34.195@KRBTEST.COM'} at 127.12.34.195:48735
I20260926 15:08:56.589774 12427 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8uoJGv/build/asan/bin/kudu as pid 14526
I20260926 15:08:56.589905 12427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435318278681-12427-0/minicluster-data/ts-2/wal/instance
I20260926 15:08:56.590301 12427 external_mini_cluster.cc:1428] Setting key e58d694a0f791e46c10601161bd53416
I20260926 15:08:56.593006 12427 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 15:08:56 dist-test-slave-865v krb5kdc[14131](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435336, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 15:08:56 dist-test-slave-865v krb5kdc[14131](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435336, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.34.254@KRBTEST.COM
I20260926 15:08:56.636410 14271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:08:56.623244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46522 (local address 127.12.34.254:45313)
0926 15:08:56.623465 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0926 15:08:56.623473 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 15:08:56.623741 (+ 268us) server_negotiation.cc:407] Connection header received
0926 15:08:56.623947 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:08:56.623951 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:08:56.624033 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:08:56.624136 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:08:56.625520 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:08:56.626328 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:08:56.627525 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:08:56.627924 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:08:56.628101 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:08:56.631705 (+ 3604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:08:56.631710 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:08:56.631713 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:08:56.631740 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:08:56.634046 (+ 2306us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:08:56.634755 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:08:56.634759 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:08:56.634761 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:08:56.634873 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:08:56.635247 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:08:56.635250 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:08:56.635252 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:08:56.635493 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:08:56.635710 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0926 15:08:56.635977 (+ 267us) server_negotiation.cc:299] Negotiation successful
0926 15:08:56.636170 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 15:08:56.640112 14175 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46522:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 15:08:56.647184 14175 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 15:08:56.681545 14678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:08:56.658132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.195:39017 (local address 127.0.0.1:36840)
0926 15:08:56.668259 (+ 10127us) negotiation.cc:107] Waiting for socket to connect
0926 15:08:56.668326 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0926 15:08:56.668533 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:08:56.672315 (+ 3782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:08:56.672327 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:08:56.672368 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:08:56.672944 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:08:56.672960 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:08:56.676218 (+ 3258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:08:56.676224 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:08:56.678525 (+ 2301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:08:56.678535 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:08:56.680578 (+ 2043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:08:56.680584 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:08:56.680795 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:08:56.680844 (+ 49us) client_negotiation.cc:769] Sending connection context
0926 15:08:56.680972 (+ 128us) client_negotiation.cc:240] Negotiation successful
0926 15:08:56.681173 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9780,"thread_start_us":17642,"threads_started":1}
I20260926 15:08:56.681545 14675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:08:56.658402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36840 (local address 127.12.34.195:39017)
0926 15:08:56.671157 (+ 12755us) server_negotiation.cc:206] Beginning negotiation
0926 15:08:56.671168 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 15:08:56.671214 (+ 46us) server_negotiation.cc:407] Connection header received
0926 15:08:56.671336 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:08:56.671348 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:08:56.671691 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:08:56.671920 (+ 229us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:08:56.673468 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:08:56.675522 (+ 2054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:08:56.678684 (+ 3162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:08:56.680359 (+ 1675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:08:56.680495 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:08:56.680734 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0926 15:08:56.681034 (+ 300us) server_negotiation.cc:299] Negotiation successful
0926 15:08:56.681210 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12548,"thread_start_us":14753,"threads_started":1}
I20260926 15:08:56.684268 14679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:08:56.660437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37124 (local address 127.12.34.194:41037)
0926 15:08:56.669634 (+ 9197us) server_negotiation.cc:206] Beginning negotiation
0926 15:08:56.669647 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 15:08:56.676335 (+ 6688us) server_negotiation.cc:407] Connection header received
0926 15:08:56.676596 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:08:56.676603 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:08:56.676790 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:08:56.676939 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:08:56.678009 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:08:56.679465 (+ 1456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:08:56.681996 (+ 2531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:08:56.683259 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:08:56.683430 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:08:56.683778 (+ 348us) server_negotiation.cc:1035] Waiting for connection context
0926 15:08:56.683945 (+ 167us) server_negotiation.cc:299] Negotiation successful
0926 15:08:56.684058 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8540,"thread_start_us":8373,"threads_started":1}
I20260926 15:08:56.684392 14676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:08:56.659840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.194:41037 (local address 127.0.0.1:37124)
0926 15:08:56.676223 (+ 16383us) negotiation.cc:107] Waiting for socket to connect
0926 15:08:56.676243 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 15:08:56.676434 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:08:56.677263 (+ 829us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:08:56.677271 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:08:56.677296 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:08:56.677797 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:08:56.677808 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:08:56.679633 (+ 1825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:08:56.679642 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:08:56.681860 (+ 2218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:08:56.681868 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:08:56.683463 (+ 1595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:08:56.683469 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:08:56.683781 (+ 312us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:08:56.683812 (+ 31us) client_negotiation.cc:769] Sending connection context
0926 15:08:56.683906 (+ 94us) client_negotiation.cc:240] Negotiation successful
0926 15:08:56.684032 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16268,"mutex_wait_us":2,"thread_start_us":7909,"threads_started":1}
I20260926 15:08:56.686406 14674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:08:56.658481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.193:45693 (local address 127.0.0.1:42090)
0926 15:08:56.676093 (+ 17612us) negotiation.cc:107] Waiting for socket to connect
0926 15:08:56.676126 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0926 15:08:56.676252 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:08:56.677174 (+ 922us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:08:56.677182 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:08:56.677209 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:08:56.677603 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:08:56.677615 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:08:56.679464 (+ 1849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:08:56.679470 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:08:56.681828 (+ 2358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:08:56.681848 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:08:56.685668 (+ 3820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:08:56.685674 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:08:56.685848 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:08:56.685885 (+ 37us) client_negotiation.cc:769] Sending connection context
0926 15:08:56.685973 (+ 88us) client_negotiation.cc:240] Negotiation successful
0926 15:08:56.686093 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17449,"thread_start_us":17472,"threads_started":1}
I20260926 15:08:56.686477 14677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:08:56.658719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42090 (local address 127.12.34.193:45693)
0926 15:08:56.675618 (+ 16899us) server_negotiation.cc:206] Beginning negotiation
0926 15:08:56.675628 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 15:08:56.676375 (+ 747us) server_negotiation.cc:407] Connection header received
0926 15:08:56.676476 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:08:56.676486 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:08:56.676744 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:08:56.676935 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:08:56.677773 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:08:56.679304 (+ 1531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:08:56.682064 (+ 2760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:08:56.685460 (+ 3396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:08:56.685618 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:08:56.685785 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0926 15:08:56.686054 (+ 269us) server_negotiation.cc:299] Negotiation successful
0926 15:08:56.686212 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16738,"thread_start_us":16897,"threads_started":1}
I20260926 15:08:56.685927 14591 tablet_service.cc:1467] Processing CreateTablet for tablet 2806d4989e1d453698a68917849b10fe (DEFAULT_TABLE table=test-table [id=1430f2be6bc94f85991e8d3fa3a24b2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:08:56.687638 14591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2806d4989e1d453698a68917849b10fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:08:56.687445 14454 tablet_service.cc:1467] Processing CreateTablet for tablet 2806d4989e1d453698a68917849b10fe (DEFAULT_TABLE table=test-table [id=1430f2be6bc94f85991e8d3fa3a24b2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:08:56.688656 14454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2806d4989e1d453698a68917849b10fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:08:56.690156 14313 tablet_service.cc:1467] Processing CreateTablet for tablet 2806d4989e1d453698a68917849b10fe (DEFAULT_TABLE table=test-table [id=1430f2be6bc94f85991e8d3fa3a24b2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:08:56.691435 14313 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2806d4989e1d453698a68917849b10fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:08:56.698727 14680 tablet_bootstrap.cc:492] T 2806d4989e1d453698a68917849b10fe P 8b4bb0b1671444b7b6c917963e957b6d: Bootstrap starting.
I20260926 15:08:56.702296 14681 tablet_bootstrap.cc:492] T 2806d4989e1d453698a68917849b10fe P 09ca737613794860ac9e00105c0b5e73: Bootstrap starting.
I20260926 15:08:56.702469 14680 tablet_bootstrap.cc:654] T 2806d4989e1d453698a68917849b10fe P 8b4bb0b1671444b7b6c917963e957b6d: Neither blocks nor log segments found. Creating new log.
I20260926 15:08:56.703740 14680 log.cc:824] T 2806d4989e1d453698a68917849b10fe P 8b4bb0b1671444b7b6c917963e957b6d: Log is configured to *not* fsync() on all Append() calls
I20260926 15:08:56.705698 14681 tablet_bootstrap.cc:654] T 2806d4989e1d453698a68917849b10fe P 09ca737613794860ac9e00105c0b5e73: Neither blocks nor log segments found. Creating new log.
I20260926 15:08:56.706866 14680 tablet_bootstrap.cc:492] T 2806d4989e1d453698a68917849b10fe P 8b4bb0b1671444b7b6c917963e957b6d: No bootstrap required, opened a new log
I20260926 15:08:56.707132 14680 ts_tablet_manager.cc:1397] T 2806d4989e1d453698a68917849b10fe P 8b4bb0b1671444b7b6c917963e957b6d: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260926 15:08:56.706903 14681 log.cc:824] T 2806d4989e1d453698a68917849b10fe P 09ca737613794860ac9e00105c0b5e73: Log is configured to *not* fsync() on all Append() calls
I20260926 15:08:56.708003 14683 tablet_bootstrap.cc:492] T 2806d4989e1d453698a68917849b10fe P 26a4aa858ba14954b2e505694f779a86: Bootstrap starting.
I20260926 15:08:56.709806 14681 tablet_bootstrap.cc:492] T 2806d4989e1d453698a68917849b10fe P 09ca737613794860ac9e00105c0b5e73: No bootstrap required, opened a new log
I20260926 15:08:56.710050 14681 ts_tablet_manager.cc:1397] T 2806d4989e1d453698a68917849b10fe P 09ca737613794860ac9e00105c0b5e73: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260926 15:08:56.711323 14683 tablet_bootstrap.cc:654] T 2806d4989e1d453698a68917849b10fe P 26a4aa858ba14954b2e505694f779a86: Neither blocks nor log segments found. Creating new log.
I20260926 15:08:56.712447 14683 log.cc:824] T 2806d4989e1d453698a68917849b10fe P 26a4aa858ba14954b2e505694f779a86: Log is configured to *not* fsync() on all Append() calls
I20260926 15:08:56.715242 14683 tablet_bootstrap.cc:492] T 2806d4989e1d453698a68917849b10fe P 26a4aa858ba14954b2e505694f779a86: No bootstrap required, opened a new log
I20260926 15:08:56.715550 14683 ts_tablet_manager.cc:1397] T 2806d4989e1d453698a68917849b10fe P 26a4aa858ba14954b2e505694f779a86: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.007s
I20260926 15:08:56.717468 14681 raft_consensus.cc:348] T 2806d4989e1d453698a68917849b10fe P 09ca737613794860ac9e00105c0b5e73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b4bb0b1671444b7b6c917963e957b6d" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 39017 } } peers { permanent_uuid: "26a4aa858ba14954b2e505694f779a86" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 45693 } } peers { permanent_uuid: "09ca737613794860ac9e00105c0b5e73" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 41037 } }
I20260926 15:08:56.717406 14680 raft_consensus.cc:348] T 2806d4989e1d453698a68917849b10fe P 8b4bb0b1671444b7b6c917963e957b6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b4bb0b1671444b7b6c917963e957b6d" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 39017 } } peers { permanent_uuid: "26a4aa858ba14954b2e505694f779a86" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 45693 } } peers { permanent_uuid: "09ca737613794860ac9e00105c0b5e73" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 41037 } }
I20260926 15:08:56.717860 14681 raft_consensus.cc:374] T 2806d4989e1d453698a68917849b10fe P 09ca737613794860ac9e00105c0b5e73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:08:56.717869 14680 raft_consensus.cc:374] T 2806d4989e1d453698a68917849b10fe P 8b4bb0b1671444b7b6c917963e957b6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:08:56.717955 14680 raft_consensus.cc:729] T 2806d4989e1d453698a68917849b10fe P 8b4bb0b1671444b7b6c917963e957b6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b4bb0b1671444b7b6c917963e957b6d, State: Initialized, Role: FOLLOWER
I20260926 15:08:56.717983 14681 raft_consensus.cc:729] T 2806d4989e1d453698a68917849b10fe P 09ca737613794860ac9e00105c0b5e73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09ca737613794860ac9e00105c0b5e73, State: Initialized, Role: FOLLOWER
I20260926 15:08:56.718611 14681 consensus_queue.cc:260] T 2806d4989e1d453698a68917849b10fe P 09ca737613794860ac9e00105c0b5e73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b4bb0b1671444b7b6c917963e957b6d" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 39017 } } peers { permanent_uuid: "26a4aa858ba14954b2e505694f779a86" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 45693 } } peers { permanent_uuid: "09ca737613794860ac9e00105c0b5e73" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 41037 } }
I20260926 15:08:56.718566 14680 consensus_queue.cc:260] T 2806d4989e1d453698a68917849b10fe P 8b4bb0b1671444b7b6c917963e957b6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b4bb0b1671444b7b6c917963e957b6d" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 39017 } } peers { permanent_uuid: "26a4aa858ba14954b2e505694f779a86" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 45693 } } peers { permanent_uuid: "09ca737613794860ac9e00105c0b5e73" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 41037 } }
I20260926 15:08:56.719748 14522 heartbeater.cc:503] Master 127.12.34.254:45313 was elected leader, sending a full tablet report...
I20260926 15:08:56.720228 14681 ts_tablet_manager.cc:1428] T 2806d4989e1d453698a68917849b10fe P 09ca737613794860ac9e00105c0b5e73: Time spent starting tablet: real 0.010s user 0.009s sys 0.000s
I20260926 15:08:56.723918 14661 heartbeater.cc:503] Master 127.12.34.254:45313 was elected leader, sending a full tablet report...
I20260926 15:08:56.727377 14680 ts_tablet_manager.cc:1428] T 2806d4989e1d453698a68917849b10fe P 8b4bb0b1671444b7b6c917963e957b6d: Time spent starting tablet: real 0.020s user 0.013s sys 0.004s
I20260926 15:08:56.727876 14683 raft_consensus.cc:348] T 2806d4989e1d453698a68917849b10fe P 26a4aa858ba14954b2e505694f779a86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b4bb0b1671444b7b6c917963e957b6d" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 39017 } } peers { permanent_uuid: "26a4aa858ba14954b2e505694f779a86" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 45693 } } peers { permanent_uuid: "09ca737613794860ac9e00105c0b5e73" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 41037 } }
I20260926 15:08:56.728266 14683 raft_consensus.cc:374] T 2806d4989e1d453698a68917849b10fe P 26a4aa858ba14954b2e505694f779a86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:08:56.728390 14683 raft_consensus.cc:729] T 2806d4989e1d453698a68917849b10fe P 26a4aa858ba14954b2e505694f779a86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26a4aa858ba14954b2e505694f779a86, State: Initialized, Role: FOLLOWER
I20260926 15:08:56.729113 14683 consensus_queue.cc:260] T 2806d4989e1d453698a68917849b10fe P 26a4aa858ba14954b2e505694f779a86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b4bb0b1671444b7b6c917963e957b6d" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 39017 } } peers { permanent_uuid: "26a4aa858ba14954b2e505694f779a86" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 45693 } } peers { permanent_uuid: "09ca737613794860ac9e00105c0b5e73" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 41037 } }
I20260926 15:08:56.730489 14683 ts_tablet_manager.cc:1428] T 2806d4989e1d453698a68917849b10fe P 26a4aa858ba14954b2e505694f779a86: Time spent starting tablet: real 0.014s user 0.012s sys 0.000s
W20260926 15:08:56.804420 14384 tablet.cc:2369] T 2806d4989e1d453698a68917849b10fe P 26a4aa858ba14954b2e505694f779a86: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 15:08:56.810204 14523 tablet.cc:2369] T 2806d4989e1d453698a68917849b10fe P 09ca737613794860ac9e00105c0b5e73: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 15:08:56.832653 14662 tablet.cc:2369] T 2806d4989e1d453698a68917849b10fe P 8b4bb0b1671444b7b6c917963e957b6d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 15:08:56.962169 14686 raft_consensus.cc:482] T 2806d4989e1d453698a68917849b10fe P 09ca737613794860ac9e00105c0b5e73 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 15:08:56.962558 14686 raft_consensus.cc:504] T 2806d4989e1d453698a68917849b10fe P 09ca737613794860ac9e00105c0b5e73 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b4bb0b1671444b7b6c917963e957b6d" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 39017 } } peers { permanent_uuid: "26a4aa858ba14954b2e505694f779a86" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 45693 } } peers { permanent_uuid: "09ca737613794860ac9e00105c0b5e73" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 41037 } }
I20260926 15:08:56.964409 14686 leader_election.cc:288] T 2806d4989e1d453698a68917849b10fe P 09ca737613794860ac9e00105c0b5e73 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8b4bb0b1671444b7b6c917963e957b6d (127.12.34.195:39017), 26a4aa858ba14954b2e505694f779a86 (127.12.34.193:45693)
I20260926 15:08:56.972183 14677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:08:56.964992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.194:47201 (local address 127.12.34.193:45693)
0926 15:08:56.965272 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0926 15:08:56.965281 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 15:08:56.966101 (+ 820us) server_negotiation.cc:407] Connection header received
0926 15:08:56.966363 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:08:56.966370 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:08:56.966461 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:08:56.966567 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:08:56.967208 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:08:56.967904 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:08:56.970399 (+ 2495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:08:56.971174 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:08:56.971310 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:08:56.971397 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 15:08:56.971757 (+ 360us) server_negotiation.cc:299] Negotiation successful
0926 15:08:56.971913 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 15:08:56.972292 14675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:08:56.964887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.194:35485 (local address 127.12.34.195:39017)
0926 15:08:56.965147 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0926 15:08:56.965157 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 15:08:56.965987 (+ 830us) server_negotiation.cc:407] Connection header received
0926 15:08:56.966199 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:08:56.966206 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:08:56.966298 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:08:56.966391 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:08:56.967054 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:08:56.967891 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:08:56.970294 (+ 2403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:08:56.971213 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:08:56.971434 (+ 221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:08:56.971511 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 15:08:56.971897 (+ 386us) server_negotiation.cc:299] Negotiation successful
0926 15:08:56.972040 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260926 15:08:56.972323 14689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:08:56.964850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.193:45693 (local address 127.12.34.194:47201)
0926 15:08:56.965985 (+ 1135us) negotiation.cc:107] Waiting for socket to connect
0926 15:08:56.966006 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 15:08:56.966219 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:08:56.966593 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:08:56.966596 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:08:56.966620 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:08:56.967064 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:08:56.967076 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:08:56.968076 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:08:56.968082 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:08:56.970246 (+ 2164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:08:56.970263 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:08:56.971405 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:08:56.971413 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:08:56.971611 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:08:56.971637 (+ 26us) client_negotiation.cc:769] Sending connection context
0926 15:08:56.971715 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 15:08:56.971851 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":967,"mutex_wait_us":24,"thread_start_us":653,"threads_started":1}
I20260926 15:08:56.972693 14690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:08:56.964798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.195:39017 (local address 127.12.34.194:35485)
0926 15:08:56.965850 (+ 1052us) negotiation.cc:107] Waiting for socket to connect
0926 15:08:56.965866 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 15:08:56.966034 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:08:56.966411 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:08:56.966415 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:08:56.966438 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:08:56.966891 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:08:56.966904 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:08:56.968025 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:08:56.968031 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:08:56.970147 (+ 2116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:08:56.970158 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:08:56.971481 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:08:56.971487 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:08:56.971671 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:08:56.971713 (+ 42us) client_negotiation.cc:769] Sending connection context
0926 15:08:56.971807 (+ 94us) client_negotiation.cc:240] Negotiation successful
0926 15:08:56.971921 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":910,"mutex_wait_us":4,"thread_start_us":577,"threads_started":1}
I20260926 15:08:56.973701 14616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2806d4989e1d453698a68917849b10fe" candidate_uuid: "09ca737613794860ac9e00105c0b5e73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b4bb0b1671444b7b6c917963e957b6d" is_pre_election: true
I20260926 15:08:56.973644 14333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2806d4989e1d453698a68917849b10fe" candidate_uuid: "09ca737613794860ac9e00105c0b5e73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26a4aa858ba14954b2e505694f779a86" is_pre_election: true
I20260926 15:08:56.974239 14616 raft_consensus.cc:2454] T 2806d4989e1d453698a68917849b10fe P 8b4bb0b1671444b7b6c917963e957b6d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09ca737613794860ac9e00105c0b5e73 in term 0.
I20260926 15:08:56.974239 14333 raft_consensus.cc:2454] T 2806d4989e1d453698a68917849b10fe P 26a4aa858ba14954b2e505694f779a86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09ca737613794860ac9e00105c0b5e73 in term 0.
I20260926 15:08:56.974965 14403 leader_election.cc:302] T 2806d4989e1d453698a68917849b10fe P 09ca737613794860ac9e00105c0b5e73 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 09ca737613794860ac9e00105c0b5e73, 26a4aa858ba14954b2e505694f779a86, 8b4bb0b1671444b7b6c917963e957b6d; no voters:
I20260926 15:08:56.975373 14686 raft_consensus.cc:2781] T 2806d4989e1d453698a68917849b10fe P 09ca737613794860ac9e00105c0b5e73 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 15:08:56.975533 14686 raft_consensus.cc:482] T 2806d4989e1d453698a68917849b10fe P 09ca737613794860ac9e00105c0b5e73 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 15:08:56.975646 14686 raft_consensus.cc:3037] T 2806d4989e1d453698a68917849b10fe P 09ca737613794860ac9e00105c0b5e73 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:08:56.977980 14686 raft_consensus.cc:504] T 2806d4989e1d453698a68917849b10fe P 09ca737613794860ac9e00105c0b5e73 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b4bb0b1671444b7b6c917963e957b6d" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 39017 } } peers { permanent_uuid: "26a4aa858ba14954b2e505694f779a86" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 45693 } } peers { permanent_uuid: "09ca737613794860ac9e00105c0b5e73" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 41037 } }
I20260926 15:08:56.978646 14686 leader_election.cc:288] T 2806d4989e1d453698a68917849b10fe P 09ca737613794860ac9e00105c0b5e73 [CANDIDATE]: Term 1 election: Requested vote from peers 8b4bb0b1671444b7b6c917963e957b6d (127.12.34.195:39017), 26a4aa858ba14954b2e505694f779a86 (127.12.34.193:45693)
I20260926 15:08:56.979033 14616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2806d4989e1d453698a68917849b10fe" candidate_uuid: "09ca737613794860ac9e00105c0b5e73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b4bb0b1671444b7b6c917963e957b6d"
I20260926 15:08:56.979120 14333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2806d4989e1d453698a68917849b10fe" candidate_uuid: "09ca737613794860ac9e00105c0b5e73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26a4aa858ba14954b2e505694f779a86"
I20260926 15:08:56.979259 14616 raft_consensus.cc:3037] T 2806d4989e1d453698a68917849b10fe P 8b4bb0b1671444b7b6c917963e957b6d [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:08:56.979303 14333 raft_consensus.cc:3037] T 2806d4989e1d453698a68917849b10fe P 26a4aa858ba14954b2e505694f779a86 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:08:56.982679 14616 raft_consensus.cc:2454] T 2806d4989e1d453698a68917849b10fe P 8b4bb0b1671444b7b6c917963e957b6d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09ca737613794860ac9e00105c0b5e73 in term 1.
I20260926 15:08:56.982703 14333 raft_consensus.cc:2454] T 2806d4989e1d453698a68917849b10fe P 26a4aa858ba14954b2e505694f779a86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09ca737613794860ac9e00105c0b5e73 in term 1.
I20260926 15:08:56.983124 14404 leader_election.cc:302] T 2806d4989e1d453698a68917849b10fe P 09ca737613794860ac9e00105c0b5e73 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 09ca737613794860ac9e00105c0b5e73, 26a4aa858ba14954b2e505694f779a86, 8b4bb0b1671444b7b6c917963e957b6d; no voters:
I20260926 15:08:56.983446 14686 raft_consensus.cc:2781] T 2806d4989e1d453698a68917849b10fe P 09ca737613794860ac9e00105c0b5e73 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:08:56.983973 14686 raft_consensus.cc:686] T 2806d4989e1d453698a68917849b10fe P 09ca737613794860ac9e00105c0b5e73 [term 1 LEADER]: Becoming Leader. State: Replica: 09ca737613794860ac9e00105c0b5e73, State: Running, Role: LEADER
I20260926 15:08:56.984587 14686 consensus_queue.cc:237] T 2806d4989e1d453698a68917849b10fe P 09ca737613794860ac9e00105c0b5e73 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b4bb0b1671444b7b6c917963e957b6d" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 39017 } } peers { permanent_uuid: "26a4aa858ba14954b2e505694f779a86" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 45693 } } peers { permanent_uuid: "09ca737613794860ac9e00105c0b5e73" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 41037 } }
I20260926 15:08:56.989696 14175 catalog_manager.cc:5495] T 2806d4989e1d453698a68917849b10fe P 09ca737613794860ac9e00105c0b5e73 reported cstate change: term changed from 0 to 1, leader changed from <none> to 09ca737613794860ac9e00105c0b5e73 (127.12.34.194). New cstate: current_term: 1 leader_uuid: "09ca737613794860ac9e00105c0b5e73" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b4bb0b1671444b7b6c917963e957b6d" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 39017 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26a4aa858ba14954b2e505694f779a86" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 45693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09ca737613794860ac9e00105c0b5e73" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 41037 } health_report { overall_health: HEALTHY } } }
Sep 26 15:08:57 dist-test-slave-865v krb5kdc[14131](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435334, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.254@KRBTEST.COM for kudu/127.12.34.254@KRBTEST.COM
I20260926 15:08:57.087414 14271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:08:57.071975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46530 (local address 127.12.34.254:45313)
0926 15:08:57.072168 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0926 15:08:57.072173 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:08:57.072864 (+ 691us) server_negotiation.cc:407] Connection header received
0926 15:08:57.073049 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:08:57.073056 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:08:57.073182 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:08:57.073278 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:08:57.074310 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:08:57.075048 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:08:57.076203 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:08:57.076655 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:08:57.076785 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:08:57.080348 (+ 3563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:08:57.080355 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:08:57.080359 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:08:57.080407 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:08:57.083864 (+ 3457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:08:57.084996 (+ 1132us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:08:57.085002 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:08:57.085004 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:08:57.085287 (+ 283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:08:57.085752 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:08:57.085755 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:08:57.085757 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:08:57.086035 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:08:57.086157 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0926 15:08:57.086905 (+ 748us) server_negotiation.cc:299] Negotiation successful
0926 15:08:57.087104 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 15:08:57.087432 14698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:08:57.072002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:45313 (local address 127.0.0.1:46530)
0926 15:08:57.072766 (+ 764us) negotiation.cc:107] Waiting for socket to connect
0926 15:08:57.072782 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 15:08:57.072939 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:08:57.073297 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:08:57.073301 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:08:57.073927 (+ 626us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:08:57.074185 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:08:57.074194 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:08:57.075162 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:08:57.075169 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:08:57.076059 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:08:57.076069 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:08:57.076823 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:08:57.076829 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:08:57.076986 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:08:57.077693 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:08:57.077714 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:08:57.080067 (+ 2353us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:08:57.084096 (+ 4029us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:08:57.084119 (+ 23us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:08:57.084129 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:08:57.084787 (+ 658us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:08:57.085471 (+ 684us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:08:57.085476 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:08:57.085478 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:08:57.085645 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:08:57.086253 (+ 608us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:08:57.086263 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:08:57.086509 (+ 246us) client_negotiation.cc:769] Sending connection context
0926 15:08:57.086712 (+ 203us) client_negotiation.cc:240] Negotiation successful
0926 15:08:57.086987 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":599,"thread_start_us":430,"threads_started":1}
I20260926 15:08:57.094203 14175 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.12.34.254@KRBTEST.COM'} at 127.0.0.1:46530:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 15:08:57.095688 14175 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 15:08:57.104041 14454 tablet_service.cc:1467] Processing CreateTablet for tablet 505da9f611004dd48cfd155d5054c182 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f2e7de5b63c14ab8849664cc2ebce340]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 15:08:57.103924 14591 tablet_service.cc:1467] Processing CreateTablet for tablet 505da9f611004dd48cfd155d5054c182 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f2e7de5b63c14ab8849664cc2ebce340]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 15:08:57.103809 14313 tablet_service.cc:1467] Processing CreateTablet for tablet 505da9f611004dd48cfd155d5054c182 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f2e7de5b63c14ab8849664cc2ebce340]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 15:08:57.104792 14454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 505da9f611004dd48cfd155d5054c182. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:08:57.104799 14591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 505da9f611004dd48cfd155d5054c182. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:08:57.104799 14313 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 505da9f611004dd48cfd155d5054c182. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:08:57.110257 14681 tablet_bootstrap.cc:492] T 505da9f611004dd48cfd155d5054c182 P 09ca737613794860ac9e00105c0b5e73: Bootstrap starting.
I20260926 15:08:57.111562 14681 tablet_bootstrap.cc:654] T 505da9f611004dd48cfd155d5054c182 P 09ca737613794860ac9e00105c0b5e73: Neither blocks nor log segments found. Creating new log.
I20260926 15:08:57.111771 14680 tablet_bootstrap.cc:492] T 505da9f611004dd48cfd155d5054c182 P 8b4bb0b1671444b7b6c917963e957b6d: Bootstrap starting.
I20260926 15:08:57.112344 14683 tablet_bootstrap.cc:492] T 505da9f611004dd48cfd155d5054c182 P 26a4aa858ba14954b2e505694f779a86: Bootstrap starting.
I20260926 15:08:57.113694 14683 tablet_bootstrap.cc:654] T 505da9f611004dd48cfd155d5054c182 P 26a4aa858ba14954b2e505694f779a86: Neither blocks nor log segments found. Creating new log.
I20260926 15:08:57.113740 14680 tablet_bootstrap.cc:654] T 505da9f611004dd48cfd155d5054c182 P 8b4bb0b1671444b7b6c917963e957b6d: Neither blocks nor log segments found. Creating new log.
I20260926 15:08:57.115759 14683 tablet_bootstrap.cc:492] T 505da9f611004dd48cfd155d5054c182 P 26a4aa858ba14954b2e505694f779a86: No bootstrap required, opened a new log
I20260926 15:08:57.115842 14681 tablet_bootstrap.cc:492] T 505da9f611004dd48cfd155d5054c182 P 09ca737613794860ac9e00105c0b5e73: No bootstrap required, opened a new log
I20260926 15:08:57.115899 14683 ts_tablet_manager.cc:1397] T 505da9f611004dd48cfd155d5054c182 P 26a4aa858ba14954b2e505694f779a86: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260926 15:08:57.115967 14681 ts_tablet_manager.cc:1397] T 505da9f611004dd48cfd155d5054c182 P 09ca737613794860ac9e00105c0b5e73: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.003s
I20260926 15:08:57.116024 14680 tablet_bootstrap.cc:492] T 505da9f611004dd48cfd155d5054c182 P 8b4bb0b1671444b7b6c917963e957b6d: No bootstrap required, opened a new log
I20260926 15:08:57.116146 14680 ts_tablet_manager.cc:1397] T 505da9f611004dd48cfd155d5054c182 P 8b4bb0b1671444b7b6c917963e957b6d: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260926 15:08:57.116622 14683 raft_consensus.cc:348] T 505da9f611004dd48cfd155d5054c182 P 26a4aa858ba14954b2e505694f779a86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26a4aa858ba14954b2e505694f779a86" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 45693 } } peers { permanent_uuid: "09ca737613794860ac9e00105c0b5e73" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 41037 } } peers { permanent_uuid: "8b4bb0b1671444b7b6c917963e957b6d" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 39017 } }
I20260926 15:08:57.116832 14683 raft_consensus.cc:374] T 505da9f611004dd48cfd155d5054c182 P 26a4aa858ba14954b2e505694f779a86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:08:57.116711 14681 raft_consensus.cc:348] T 505da9f611004dd48cfd155d5054c182 P 09ca737613794860ac9e00105c0b5e73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26a4aa858ba14954b2e505694f779a86" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 45693 } } peers { permanent_uuid: "09ca737613794860ac9e00105c0b5e73" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 41037 } } peers { permanent_uuid: "8b4bb0b1671444b7b6c917963e957b6d" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 39017 } }
I20260926 15:08:57.116935 14681 raft_consensus.cc:374] T 505da9f611004dd48cfd155d5054c182 P 09ca737613794860ac9e00105c0b5e73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:08:57.116928 14683 raft_consensus.cc:729] T 505da9f611004dd48cfd155d5054c182 P 26a4aa858ba14954b2e505694f779a86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26a4aa858ba14954b2e505694f779a86, State: Initialized, Role: FOLLOWER
I20260926 15:08:57.116806 14680 raft_consensus.cc:348] T 505da9f611004dd48cfd155d5054c182 P 8b4bb0b1671444b7b6c917963e957b6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26a4aa858ba14954b2e505694f779a86" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 45693 } } peers { permanent_uuid: "09ca737613794860ac9e00105c0b5e73" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 41037 } } peers { permanent_uuid: "8b4bb0b1671444b7b6c917963e957b6d" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 39017 } }
I20260926 15:08:57.117033 14681 raft_consensus.cc:729] T 505da9f611004dd48cfd155d5054c182 P 09ca737613794860ac9e00105c0b5e73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09ca737613794860ac9e00105c0b5e73, State: Initialized, Role: FOLLOWER
I20260926 15:08:57.117048 14680 raft_consensus.cc:374] T 505da9f611004dd48cfd155d5054c182 P 8b4bb0b1671444b7b6c917963e957b6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:08:57.117167 14680 raft_consensus.cc:729] T 505da9f611004dd48cfd155d5054c182 P 8b4bb0b1671444b7b6c917963e957b6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b4bb0b1671444b7b6c917963e957b6d, State: Initialized, Role: FOLLOWER
I20260926 15:08:57.117220 14683 consensus_queue.cc:260] T 505da9f611004dd48cfd155d5054c182 P 26a4aa858ba14954b2e505694f779a86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26a4aa858ba14954b2e505694f779a86" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 45693 } } peers { permanent_uuid: "09ca737613794860ac9e00105c0b5e73" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 41037 } } peers { permanent_uuid: "8b4bb0b1671444b7b6c917963e957b6d" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 39017 } }
I20260926 15:08:57.117293 14681 consensus_queue.cc:260] T 505da9f611004dd48cfd155d5054c182 P 09ca737613794860ac9e00105c0b5e73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26a4aa858ba14954b2e505694f779a86" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 45693 } } peers { permanent_uuid: "09ca737613794860ac9e00105c0b5e73" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 41037 } } peers { permanent_uuid: "8b4bb0b1671444b7b6c917963e957b6d" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 39017 } }
I20260926 15:08:57.117375 14680 consensus_queue.cc:260] T 505da9f611004dd48cfd155d5054c182 P 8b4bb0b1671444b7b6c917963e957b6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26a4aa858ba14954b2e505694f779a86" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 45693 } } peers { permanent_uuid: "09ca737613794860ac9e00105c0b5e73" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 41037 } } peers { permanent_uuid: "8b4bb0b1671444b7b6c917963e957b6d" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 39017 } }
I20260926 15:08:57.117806 14683 ts_tablet_manager.cc:1428] T 505da9f611004dd48cfd155d5054c182 P 26a4aa858ba14954b2e505694f779a86: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260926 15:08:57.117690 14686 tablet_replica.cc:439] T 505da9f611004dd48cfd155d5054c182 P 09ca737613794860ac9e00105c0b5e73: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26a4aa858ba14954b2e505694f779a86" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 45693 } } peers { permanent_uuid: "09ca737613794860ac9e00105c0b5e73" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 41037 } } peers { permanent_uuid: "8b4bb0b1671444b7b6c917963e957b6d" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 39017 } } }
I20260926 15:08:57.117908 14681 ts_tablet_manager.cc:1428] T 505da9f611004dd48cfd155d5054c182 P 09ca737613794860ac9e00105c0b5e73: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260926 15:08:57.117682 14688 tablet_replica.cc:439] T 505da9f611004dd48cfd155d5054c182 P 26a4aa858ba14954b2e505694f779a86: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26a4aa858ba14954b2e505694f779a86" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 45693 } } peers { permanent_uuid: "09ca737613794860ac9e00105c0b5e73" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 41037 } } peers { permanent_uuid: "8b4bb0b1671444b7b6c917963e957b6d" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 39017 } } }
I20260926 15:08:57.117980 14686 tablet_replica.cc:442] T 505da9f611004dd48cfd155d5054c182 P 09ca737613794860ac9e00105c0b5e73: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 15:08:57.118041 14688 tablet_replica.cc:442] T 505da9f611004dd48cfd155d5054c182 P 26a4aa858ba14954b2e505694f779a86: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 15:08:57.118422 14680 ts_tablet_manager.cc:1428] T 505da9f611004dd48cfd155d5054c182 P 8b4bb0b1671444b7b6c917963e957b6d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 15:08:57.118637 14687 tablet_replica.cc:439] T 505da9f611004dd48cfd155d5054c182 P 8b4bb0b1671444b7b6c917963e957b6d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26a4aa858ba14954b2e505694f779a86" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 45693 } } peers { permanent_uuid: "09ca737613794860ac9e00105c0b5e73" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 41037 } } peers { permanent_uuid: "8b4bb0b1671444b7b6c917963e957b6d" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 39017 } } }
I20260926 15:08:57.118911 14687 tablet_replica.cc:442] T 505da9f611004dd48cfd155d5054c182 P 8b4bb0b1671444b7b6c917963e957b6d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 15:08:57.342278 14687 raft_consensus.cc:482] T 505da9f611004dd48cfd155d5054c182 P 8b4bb0b1671444b7b6c917963e957b6d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 15:08:57.342485 14687 raft_consensus.cc:504] T 505da9f611004dd48cfd155d5054c182 P 8b4bb0b1671444b7b6c917963e957b6d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26a4aa858ba14954b2e505694f779a86" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 45693 } } peers { permanent_uuid: "09ca737613794860ac9e00105c0b5e73" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 41037 } } peers { permanent_uuid: "8b4bb0b1671444b7b6c917963e957b6d" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 39017 } }
I20260926 15:08:57.343869 14687 leader_election.cc:288] T 505da9f611004dd48cfd155d5054c182 P 8b4bb0b1671444b7b6c917963e957b6d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 26a4aa858ba14954b2e505694f779a86 (127.12.34.193:45693), 09ca737613794860ac9e00105c0b5e73 (127.12.34.194:41037)
I20260926 15:08:57.352820 14677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:08:57.344415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.195:57367 (local address 127.12.34.193:45693)
0926 15:08:57.344614 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0926 15:08:57.344622 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 15:08:57.345063 (+ 441us) server_negotiation.cc:407] Connection header received
0926 15:08:57.345304 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:08:57.345309 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:08:57.345389 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:08:57.345489 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:08:57.346192 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:08:57.347266 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:08:57.350338 (+ 3072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:08:57.351631 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:08:57.351796 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:08:57.351870 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 15:08:57.352397 (+ 527us) server_negotiation.cc:299] Negotiation successful
0926 15:08:57.352527 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260926 15:08:57.352838 14702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:08:57.344232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.195:42527 (local address 127.12.34.194:41037)
0926 15:08:57.344758 (+ 526us) server_negotiation.cc:206] Beginning negotiation
0926 15:08:57.344764 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:08:57.345180 (+ 416us) server_negotiation.cc:407] Connection header received
0926 15:08:57.345322 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:08:57.345327 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:08:57.345391 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:08:57.345519 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:08:57.346086 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:08:57.347266 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:08:57.350339 (+ 3073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:08:57.351631 (+ 1292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:08:57.351796 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:08:57.351871 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 15:08:57.352397 (+ 526us) server_negotiation.cc:299] Negotiation successful
0926 15:08:57.352559 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":264,"threads_started":1}
I20260926 15:08:57.352977 14703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:08:57.344168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.193:45693 (local address 127.12.34.195:57367)
0926 15:08:57.344953 (+ 785us) negotiation.cc:107] Waiting for socket to connect
0926 15:08:57.344974 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 15:08:57.345147 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:08:57.345514 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:08:57.345519 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:08:57.345622 (+ 103us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:08:57.346054 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:08:57.346066 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:08:57.347448 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:08:57.347454 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:08:57.350150 (+ 2696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:08:57.350167 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:08:57.351832 (+ 1665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:08:57.351841 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:08:57.352097 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:08:57.352254 (+ 157us) client_negotiation.cc:769] Sending connection context
0926 15:08:57.352355 (+ 101us) client_negotiation.cc:240] Negotiation successful
0926 15:08:57.352538 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":633,"mutex_wait_us":290,"spinlock_wait_cycles":23424,"thread_start_us":511,"threads_started":1}
I20260926 15:08:57.353232 14704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:08:57.344177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.194:41037 (local address 127.12.34.195:42527)
0926 15:08:57.345097 (+ 920us) negotiation.cc:107] Waiting for socket to connect
0926 15:08:57.345111 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 15:08:57.345208 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:08:57.345528 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:08:57.345532 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:08:57.345556 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:08:57.345905 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:08:57.345920 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:08:57.347448 (+ 1528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:08:57.347454 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:08:57.350150 (+ 2696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:08:57.350167 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:08:57.351875 (+ 1708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:08:57.351880 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:08:57.352114 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:08:57.352252 (+ 138us) client_negotiation.cc:769] Sending connection context
0926 15:08:57.352355 (+ 103us) client_negotiation.cc:240] Negotiation successful
0926 15:08:57.352538 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":776,"mutex_wait_us":211,"thread_start_us":625,"threads_started":1}
I20260926 15:08:57.353935 14333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "505da9f611004dd48cfd155d5054c182" candidate_uuid: "8b4bb0b1671444b7b6c917963e957b6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26a4aa858ba14954b2e505694f779a86" is_pre_election: true
I20260926 15:08:57.354182 14333 raft_consensus.cc:2454] T 505da9f611004dd48cfd155d5054c182 P 26a4aa858ba14954b2e505694f779a86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b4bb0b1671444b7b6c917963e957b6d in term 0.
I20260926 15:08:57.354254 14477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "505da9f611004dd48cfd155d5054c182" candidate_uuid: "8b4bb0b1671444b7b6c917963e957b6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ca737613794860ac9e00105c0b5e73" is_pre_election: true
I20260926 15:08:57.354681 14477 raft_consensus.cc:2454] T 505da9f611004dd48cfd155d5054c182 P 09ca737613794860ac9e00105c0b5e73 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b4bb0b1671444b7b6c917963e957b6d in term 0.
I20260926 15:08:57.354665 14543 leader_election.cc:302] T 505da9f611004dd48cfd155d5054c182 P 8b4bb0b1671444b7b6c917963e957b6d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26a4aa858ba14954b2e505694f779a86, 8b4bb0b1671444b7b6c917963e957b6d; no voters:
I20260926 15:08:57.355099 14687 raft_consensus.cc:2781] T 505da9f611004dd48cfd155d5054c182 P 8b4bb0b1671444b7b6c917963e957b6d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 15:08:57.355244 14687 raft_consensus.cc:482] T 505da9f611004dd48cfd155d5054c182 P 8b4bb0b1671444b7b6c917963e957b6d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 15:08:57.355357 14687 raft_consensus.cc:3037] T 505da9f611004dd48cfd155d5054c182 P 8b4bb0b1671444b7b6c917963e957b6d [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:08:57.357470 14687 raft_consensus.cc:504] T 505da9f611004dd48cfd155d5054c182 P 8b4bb0b1671444b7b6c917963e957b6d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26a4aa858ba14954b2e505694f779a86" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 45693 } } peers { permanent_uuid: "09ca737613794860ac9e00105c0b5e73" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 41037 } } peers { permanent_uuid: "8b4bb0b1671444b7b6c917963e957b6d" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 39017 } }
I20260926 15:08:57.357983 14687 leader_election.cc:288] T 505da9f611004dd48cfd155d5054c182 P 8b4bb0b1671444b7b6c917963e957b6d [CANDIDATE]: Term 1 election: Requested vote from peers 26a4aa858ba14954b2e505694f779a86 (127.12.34.193:45693), 09ca737613794860ac9e00105c0b5e73 (127.12.34.194:41037)
I20260926 15:08:57.358428 14477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "505da9f611004dd48cfd155d5054c182" candidate_uuid: "8b4bb0b1671444b7b6c917963e957b6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ca737613794860ac9e00105c0b5e73"
I20260926 15:08:57.358409 14333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "505da9f611004dd48cfd155d5054c182" candidate_uuid: "8b4bb0b1671444b7b6c917963e957b6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26a4aa858ba14954b2e505694f779a86"
I20260926 15:08:57.358613 14477 raft_consensus.cc:3037] T 505da9f611004dd48cfd155d5054c182 P 09ca737613794860ac9e00105c0b5e73 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:08:57.358642 14333 raft_consensus.cc:3037] T 505da9f611004dd48cfd155d5054c182 P 26a4aa858ba14954b2e505694f779a86 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:08:57.361528 14333 raft_consensus.cc:2454] T 505da9f611004dd48cfd155d5054c182 P 26a4aa858ba14954b2e505694f779a86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b4bb0b1671444b7b6c917963e957b6d in term 1.
I20260926 15:08:57.361618 14477 raft_consensus.cc:2454] T 505da9f611004dd48cfd155d5054c182 P 09ca737613794860ac9e00105c0b5e73 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b4bb0b1671444b7b6c917963e957b6d in term 1.
I20260926 15:08:57.361915 14543 leader_election.cc:302] T 505da9f611004dd48cfd155d5054c182 P 8b4bb0b1671444b7b6c917963e957b6d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26a4aa858ba14954b2e505694f779a86, 8b4bb0b1671444b7b6c917963e957b6d; no voters:
I20260926 15:08:57.362175 14687 raft_consensus.cc:2781] T 505da9f611004dd48cfd155d5054c182 P 8b4bb0b1671444b7b6c917963e957b6d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:08:57.362592 14687 raft_consensus.cc:686] T 505da9f611004dd48cfd155d5054c182 P 8b4bb0b1671444b7b6c917963e957b6d [term 1 LEADER]: Becoming Leader. State: Replica: 8b4bb0b1671444b7b6c917963e957b6d, State: Running, Role: LEADER
I20260926 15:08:57.363049 14687 consensus_queue.cc:237] T 505da9f611004dd48cfd155d5054c182 P 8b4bb0b1671444b7b6c917963e957b6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26a4aa858ba14954b2e505694f779a86" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 45693 } } peers { permanent_uuid: "09ca737613794860ac9e00105c0b5e73" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 41037 } } peers { permanent_uuid: "8b4bb0b1671444b7b6c917963e957b6d" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 39017 } }
I20260926 15:08:57.366427 14705 tablet_replica.cc:439] T 505da9f611004dd48cfd155d5054c182 P 8b4bb0b1671444b7b6c917963e957b6d: TxnStatusTablet state changed. Reason: New leader 8b4bb0b1671444b7b6c917963e957b6d. Latest consensus state: current_term: 1 leader_uuid: "8b4bb0b1671444b7b6c917963e957b6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26a4aa858ba14954b2e505694f779a86" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 45693 } } peers { permanent_uuid: "09ca737613794860ac9e00105c0b5e73" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 41037 } } peers { permanent_uuid: "8b4bb0b1671444b7b6c917963e957b6d" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 39017 } } }
I20260926 15:08:57.366743 14705 tablet_replica.cc:442] T 505da9f611004dd48cfd155d5054c182 P 8b4bb0b1671444b7b6c917963e957b6d: This TxnStatusTablet replica's current role is: LEADER
I20260926 15:08:57.367492 14707 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 15:08:57.367765 14707 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 15:08:57.368223 14174 catalog_manager.cc:5495] T 505da9f611004dd48cfd155d5054c182 P 8b4bb0b1671444b7b6c917963e957b6d reported cstate change: term changed from 0 to 1, leader changed from <none> to 8b4bb0b1671444b7b6c917963e957b6d (127.12.34.195). New cstate: current_term: 1 leader_uuid: "8b4bb0b1671444b7b6c917963e957b6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26a4aa858ba14954b2e505694f779a86" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 45693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09ca737613794860ac9e00105c0b5e73" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 41037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b4bb0b1671444b7b6c917963e957b6d" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 39017 } health_report { overall_health: HEALTHY } } }
I20260926 15:08:57.412307 14675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:08:57.406893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36846 (local address 127.12.34.195:39017)
0926 15:08:57.407086 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0926 15:08:57.407093 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:08:57.407237 (+ 144us) server_negotiation.cc:407] Connection header received
0926 15:08:57.407545 (+ 308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:08:57.407549 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:08:57.407626 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:08:57.407759 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 15:08:57.408096 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:08:57.408891 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:08:57.410178 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:08:57.410583 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:08:57.410704 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:08:57.411134 (+ 430us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 15:08:57.411636 (+ 502us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 15:08:57.411791 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0926 15:08:57.411847 (+ 56us) server_negotiation.cc:299] Negotiation successful
0926 15:08:57.411941 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 15:08:57.412284 14698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:08:57.406901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.195:39017 (local address 127.0.0.1:36846)
0926 15:08:57.407148 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0926 15:08:57.407172 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 15:08:57.407335 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:08:57.407774 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:08:57.407779 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:08:57.407790 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 15:08:57.407975 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:08:57.407983 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:08:57.409017 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:08:57.409025 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:08:57.410045 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:08:57.410055 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:08:57.410741 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:08:57.410747 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:08:57.410904 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:08:57.410955 (+ 51us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 15:08:57.411737 (+ 782us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 15:08:57.411756 (+ 19us) client_negotiation.cc:769] Sending connection context
0926 15:08:57.411830 (+ 74us) client_negotiation.cc:240] Negotiation successful
0926 15:08:57.411974 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85}
I20260926 15:08:57.420233 14333 raft_consensus.cc:1261] T 505da9f611004dd48cfd155d5054c182 P 26a4aa858ba14954b2e505694f779a86 [term 1 FOLLOWER]: Refusing update from remote peer 8b4bb0b1671444b7b6c917963e957b6d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:08:57.420302 14477 raft_consensus.cc:1261] T 505da9f611004dd48cfd155d5054c182 P 09ca737613794860ac9e00105c0b5e73 [term 1 FOLLOWER]: Refusing update from remote peer 8b4bb0b1671444b7b6c917963e957b6d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:08:57.421165 14705 consensus_queue.cc:1035] T 505da9f611004dd48cfd155d5054c182 P 8b4bb0b1671444b7b6c917963e957b6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "26a4aa858ba14954b2e505694f779a86" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 45693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:08:57.421469 14687 consensus_queue.cc:1035] T 505da9f611004dd48cfd155d5054c182 P 8b4bb0b1671444b7b6c917963e957b6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "09ca737613794860ac9e00105c0b5e73" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:08:57.430387 14688 tablet_replica.cc:439] T 505da9f611004dd48cfd155d5054c182 P 26a4aa858ba14954b2e505694f779a86: TxnStatusTablet state changed. Reason: New leader 8b4bb0b1671444b7b6c917963e957b6d. Latest consensus state: current_term: 1 leader_uuid: "8b4bb0b1671444b7b6c917963e957b6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26a4aa858ba14954b2e505694f779a86" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 45693 } } peers { permanent_uuid: "09ca737613794860ac9e00105c0b5e73" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 41037 } } peers { permanent_uuid: "8b4bb0b1671444b7b6c917963e957b6d" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 39017 } } }
I20260926 15:08:57.430677 14688 tablet_replica.cc:442] T 505da9f611004dd48cfd155d5054c182 P 26a4aa858ba14954b2e505694f779a86: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 15:08:57.433357 14710 mvcc.cc:205] Tried to move back new op lower bound from 7333623142063206400 to 7333623141842259968. Current Snapshot: MvccSnapshot[applied={T|T < 7333623142063206400}]
I20260926 15:08:57.433941 14687 tablet_replica.cc:439] T 505da9f611004dd48cfd155d5054c182 P 8b4bb0b1671444b7b6c917963e957b6d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8b4bb0b1671444b7b6c917963e957b6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26a4aa858ba14954b2e505694f779a86" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 45693 } } peers { permanent_uuid: "09ca737613794860ac9e00105c0b5e73" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 41037 } } peers { permanent_uuid: "8b4bb0b1671444b7b6c917963e957b6d" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 39017 } } }
I20260926 15:08:57.434186 14687 tablet_replica.cc:442] T 505da9f611004dd48cfd155d5054c182 P 8b4bb0b1671444b7b6c917963e957b6d: This TxnStatusTablet replica's current role is: LEADER
I20260926 15:08:57.445878 14686 tablet_replica.cc:439] T 505da9f611004dd48cfd155d5054c182 P 09ca737613794860ac9e00105c0b5e73: TxnStatusTablet state changed. Reason: New leader 8b4bb0b1671444b7b6c917963e957b6d. Latest consensus state: current_term: 1 leader_uuid: "8b4bb0b1671444b7b6c917963e957b6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26a4aa858ba14954b2e505694f779a86" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 45693 } } peers { permanent_uuid: "09ca737613794860ac9e00105c0b5e73" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 41037 } } peers { permanent_uuid: "8b4bb0b1671444b7b6c917963e957b6d" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 39017 } } }
I20260926 15:08:57.446241 14686 tablet_replica.cc:442] T 505da9f611004dd48cfd155d5054c182 P 09ca737613794860ac9e00105c0b5e73: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 15:08:57.448122 14705 tablet_replica.cc:439] T 505da9f611004dd48cfd155d5054c182 P 8b4bb0b1671444b7b6c917963e957b6d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8b4bb0b1671444b7b6c917963e957b6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26a4aa858ba14954b2e505694f779a86" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 45693 } } peers { permanent_uuid: "09ca737613794860ac9e00105c0b5e73" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 41037 } } peers { permanent_uuid: "8b4bb0b1671444b7b6c917963e957b6d" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 39017 } } }
I20260926 15:08:57.448721 14705 tablet_replica.cc:442] T 505da9f611004dd48cfd155d5054c182 P 8b4bb0b1671444b7b6c917963e957b6d: This TxnStatusTablet replica's current role is: LEADER
I20260926 15:08:57.449164 14702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:08:57.442173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37126 (local address 127.12.34.194:41037)
0926 15:08:57.442541 (+ 368us) server_negotiation.cc:206] Beginning negotiation
0926 15:08:57.442549 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 15:08:57.442915 (+ 366us) server_negotiation.cc:407] Connection header received
0926 15:08:57.443141 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:08:57.443146 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:08:57.443228 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:08:57.443373 (+ 145us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 15:08:57.443780 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:08:57.444500 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:08:57.446501 (+ 2001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:08:57.446959 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:08:57.447064 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:08:57.447689 (+ 625us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 15:08:57.448363 (+ 674us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 15:08:57.448464 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0926 15:08:57.448736 (+ 272us) server_negotiation.cc:299] Negotiation successful
0926 15:08:57.448816 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":201}
I20260926 15:08:57.449971 14712 mvcc.cc:205] Tried to move back new op lower bound from 7333623142063206400 to 7333623141842259968. Current Snapshot: MvccSnapshot[applied={T|T < 7333623142063206400}]
I20260926 15:08:57.452044 14688 tablet_replica.cc:439] T 505da9f611004dd48cfd155d5054c182 P 26a4aa858ba14954b2e505694f779a86: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8b4bb0b1671444b7b6c917963e957b6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26a4aa858ba14954b2e505694f779a86" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 45693 } } peers { permanent_uuid: "09ca737613794860ac9e00105c0b5e73" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 41037 } } peers { permanent_uuid: "8b4bb0b1671444b7b6c917963e957b6d" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 39017 } } }
I20260926 15:08:57.452241 14688 tablet_replica.cc:442] T 505da9f611004dd48cfd155d5054c182 P 26a4aa858ba14954b2e505694f779a86: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 15:08:57.454255 14686 tablet_replica.cc:439] T 505da9f611004dd48cfd155d5054c182 P 09ca737613794860ac9e00105c0b5e73: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8b4bb0b1671444b7b6c917963e957b6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26a4aa858ba14954b2e505694f779a86" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 45693 } } peers { permanent_uuid: "09ca737613794860ac9e00105c0b5e73" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 41037 } } peers { permanent_uuid: "8b4bb0b1671444b7b6c917963e957b6d" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 39017 } } }
I20260926 15:08:57.454537 14686 tablet_replica.cc:442] T 505da9f611004dd48cfd155d5054c182 P 09ca737613794860ac9e00105c0b5e73: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 15:08:57.466898 14675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:08:57.460718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.194:56527 (local address 127.12.34.195:39017)
0926 15:08:57.460914 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0926 15:08:57.460922 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 15:08:57.461464 (+ 542us) server_negotiation.cc:407] Connection header received
0926 15:08:57.461656 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:08:57.461660 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:08:57.461720 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:08:57.461793 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 15:08:57.462367 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:08:57.463104 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:08:57.465144 (+ 2040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:08:57.465660 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:08:57.465771 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:08:57.466269 (+ 498us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 15:08:57.466356 (+ 87us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 15:08:57.466450 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 15:08:57.466574 (+ 124us) server_negotiation.cc:299] Negotiation successful
0926 15:08:57.466630 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 15:08:57.467165 14722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:08:57.460717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.195:39017 (local address 127.12.34.194:56527)
0926 15:08:57.461364 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0926 15:08:57.461383 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 15:08:57.461522 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:08:57.461826 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:08:57.461831 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:08:57.461844 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 15:08:57.462183 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:08:57.462196 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:08:57.463232 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:08:57.463238 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:08:57.464958 (+ 1720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:08:57.464972 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:08:57.465831 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:08:57.465837 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:08:57.466011 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:08:57.466095 (+ 84us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 15:08:57.466467 (+ 372us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 15:08:57.466478 (+ 11us) client_negotiation.cc:769] Sending connection context
0926 15:08:57.466562 (+ 84us) client_negotiation.cc:240] Negotiation successful
0926 15:08:57.466691 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":382,"threads_started":1}
I20260926 15:08:57.476343 14333 raft_consensus.cc:1261] T 2806d4989e1d453698a68917849b10fe P 26a4aa858ba14954b2e505694f779a86 [term 1 FOLLOWER]: Refusing update from remote peer 09ca737613794860ac9e00105c0b5e73: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:08:57.476579 14616 raft_consensus.cc:1261] T 2806d4989e1d453698a68917849b10fe P 8b4bb0b1671444b7b6c917963e957b6d [term 1 FOLLOWER]: Refusing update from remote peer 09ca737613794860ac9e00105c0b5e73: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:08:57.476866 14691 consensus_queue.cc:1035] T 2806d4989e1d453698a68917849b10fe P 09ca737613794860ac9e00105c0b5e73 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26a4aa858ba14954b2e505694f779a86" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 45693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:08:57.477249 14686 consensus_queue.cc:1035] T 2806d4989e1d453698a68917849b10fe P 09ca737613794860ac9e00105c0b5e73 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b4bb0b1671444b7b6c917963e957b6d" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 39017 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:08:57.509917 14702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:08:57.503256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.195:46059 (local address 127.12.34.194:41037)
0926 15:08:57.503420 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0926 15:08:57.503427 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:08:57.503949 (+ 522us) server_negotiation.cc:407] Connection header received
0926 15:08:57.504070 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:08:57.504075 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:08:57.504155 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:08:57.504300 (+ 145us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 15:08:57.504970 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:08:57.506130 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:08:57.508069 (+ 1939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:08:57.508647 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:08:57.508770 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:08:57.509194 (+ 424us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 15:08:57.509315 (+ 121us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 15:08:57.509397 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 15:08:57.509535 (+ 138us) server_negotiation.cc:299] Negotiation successful
0926 15:08:57.509610 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 15:08:57.510039 14727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:08:57.503324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.194:41037 (local address 127.12.34.195:46059)
0926 15:08:57.503845 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0926 15:08:57.503859 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 15:08:57.503974 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:08:57.504370 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:08:57.504375 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:08:57.504388 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 15:08:57.504803 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:08:57.504818 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:08:57.506271 (+ 1453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:08:57.506275 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:08:57.507936 (+ 1661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:08:57.507948 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:08:57.508809 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:08:57.508832 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:08:57.508952 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:08:57.509001 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 15:08:57.509411 (+ 410us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 15:08:57.509420 (+ 9us) client_negotiation.cc:769] Sending connection context
0926 15:08:57.509506 (+ 86us) client_negotiation.cc:240] Negotiation successful
0926 15:08:57.509639 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":248,"threads_started":1}
I20260926 15:08:57.543676 14175 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46522:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 15:08:57.543962 14175 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46522:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 15:08:57.547794 14175 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 737da02a56c94512be48ba241fbb275d: Sending DeleteTablet for 3 replicas of tablet 2806d4989e1d453698a68917849b10fe
I20260926 15:08:57.548869 14591 tablet_service.cc:1514] Processing DeleteTablet for tablet 2806d4989e1d453698a68917849b10fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:08:57 UTC) from {username='kudu', principal='kudu/127.12.34.254@KRBTEST.COM'} at 127.0.0.1:36840
I20260926 15:08:57.548950 14454 tablet_service.cc:1514] Processing DeleteTablet for tablet 2806d4989e1d453698a68917849b10fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:08:57 UTC) from {username='kudu', principal='kudu/127.12.34.254@KRBTEST.COM'} at 127.0.0.1:37124
I20260926 15:08:57.550220 14313 tablet_service.cc:1514] Processing DeleteTablet for tablet 2806d4989e1d453698a68917849b10fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:08:57 UTC) from {username='kudu', principal='kudu/127.12.34.254@KRBTEST.COM'} at 127.0.0.1:42090
I20260926 15:08:57.557068 14175 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46522:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 15:08:57.559002 14175 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 15:08:57.568142 14312 tablet_service.cc:1467] Processing CreateTablet for tablet eb562d4b848d4488b3505840593c04a2 (DEFAULT_TABLE table=test-table [id=db53f1b14a6e489299fc29cff7a37838]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:08:57.568459 14452 tablet_service.cc:1467] Processing CreateTablet for tablet eb562d4b848d4488b3505840593c04a2 (DEFAULT_TABLE table=test-table [id=db53f1b14a6e489299fc29cff7a37838]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:08:57.568743 14312 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb562d4b848d4488b3505840593c04a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:08:57.569250 14452 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb562d4b848d4488b3505840593c04a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:08:57.572726 14590 tablet_service.cc:1467] Processing CreateTablet for tablet eb562d4b848d4488b3505840593c04a2 (DEFAULT_TABLE table=test-table [id=db53f1b14a6e489299fc29cff7a37838]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:08:57.573274 14590 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb562d4b848d4488b3505840593c04a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:08:57.576541 14681 tablet_bootstrap.cc:492] T eb562d4b848d4488b3505840593c04a2 P 09ca737613794860ac9e00105c0b5e73: Bootstrap starting.
I20260926 15:08:57.578284 14681 tablet_bootstrap.cc:654] T eb562d4b848d4488b3505840593c04a2 P 09ca737613794860ac9e00105c0b5e73: Neither blocks nor log segments found. Creating new log.
I20260926 15:08:57.582432 14733 tablet_replica.cc:330] T 2806d4989e1d453698a68917849b10fe P 26a4aa858ba14954b2e505694f779a86: stopping tablet replica
I20260926 15:08:57.582824 14733 raft_consensus.cc:2229] T 2806d4989e1d453698a68917849b10fe P 26a4aa858ba14954b2e505694f779a86 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 15:08:57.583210 14733 raft_consensus.cc:2258] T 2806d4989e1d453698a68917849b10fe P 26a4aa858ba14954b2e505694f779a86 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:08:57.584043 14732 tablet_replica.cc:330] T 2806d4989e1d453698a68917849b10fe P 09ca737613794860ac9e00105c0b5e73: stopping tablet replica
I20260926 15:08:57.584448 14732 raft_consensus.cc:2229] T 2806d4989e1d453698a68917849b10fe P 09ca737613794860ac9e00105c0b5e73 [term 1 LEADER]: Raft consensus shutting down.
I20260926 15:08:57.587092 14680 tablet_bootstrap.cc:492] T eb562d4b848d4488b3505840593c04a2 P 8b4bb0b1671444b7b6c917963e957b6d: Bootstrap starting.
I20260926 15:08:57.588311 14683 tablet_bootstrap.cc:492] T eb562d4b848d4488b3505840593c04a2 P 26a4aa858ba14954b2e505694f779a86: Bootstrap starting.
I20260926 15:08:57.588858 14680 tablet_bootstrap.cc:654] T eb562d4b848d4488b3505840593c04a2 P 8b4bb0b1671444b7b6c917963e957b6d: Neither blocks nor log segments found. Creating new log.
I20260926 15:08:57.587625 14732 raft_consensus.cc:2258] T 2806d4989e1d453698a68917849b10fe P 09ca737613794860ac9e00105c0b5e73 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:08:57.590147 14683 tablet_bootstrap.cc:654] T eb562d4b848d4488b3505840593c04a2 P 26a4aa858ba14954b2e505694f779a86: Neither blocks nor log segments found. Creating new log.
I20260926 15:08:57.592878 14733 ts_tablet_manager.cc:1905] T 2806d4989e1d453698a68917849b10fe P 26a4aa858ba14954b2e505694f779a86: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:08:57.595577 14681 tablet_bootstrap.cc:492] T eb562d4b848d4488b3505840593c04a2 P 09ca737613794860ac9e00105c0b5e73: No bootstrap required, opened a new log
I20260926 15:08:57.595840 14681 ts_tablet_manager.cc:1397] T eb562d4b848d4488b3505840593c04a2 P 09ca737613794860ac9e00105c0b5e73: Time spent bootstrapping tablet: real 0.019s user 0.005s sys 0.007s
I20260926 15:08:57.596081 14732 ts_tablet_manager.cc:1905] T 2806d4989e1d453698a68917849b10fe P 09ca737613794860ac9e00105c0b5e73: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:08:57.597167 14731 tablet_replica.cc:330] T 2806d4989e1d453698a68917849b10fe P 8b4bb0b1671444b7b6c917963e957b6d: stopping tablet replica
I20260926 15:08:57.598839 14680 tablet_bootstrap.cc:492] T eb562d4b848d4488b3505840593c04a2 P 8b4bb0b1671444b7b6c917963e957b6d: No bootstrap required, opened a new log
I20260926 15:08:57.599105 14680 ts_tablet_manager.cc:1397] T eb562d4b848d4488b3505840593c04a2 P 8b4bb0b1671444b7b6c917963e957b6d: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.000s
I20260926 15:08:57.599987 14680 raft_consensus.cc:348] T eb562d4b848d4488b3505840593c04a2 P 8b4bb0b1671444b7b6c917963e957b6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26a4aa858ba14954b2e505694f779a86" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 45693 } } peers { permanent_uuid: "09ca737613794860ac9e00105c0b5e73" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 41037 } } peers { permanent_uuid: "8b4bb0b1671444b7b6c917963e957b6d" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 39017 } }
I20260926 15:08:57.600315 14680 raft_consensus.cc:374] T eb562d4b848d4488b3505840593c04a2 P 8b4bb0b1671444b7b6c917963e957b6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:08:57.600445 14680 raft_consensus.cc:729] T eb562d4b848d4488b3505840593c04a2 P 8b4bb0b1671444b7b6c917963e957b6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b4bb0b1671444b7b6c917963e957b6d, State: Initialized, Role: FOLLOWER
I20260926 15:08:57.600847 14680 consensus_queue.cc:260] T eb562d4b848d4488b3505840593c04a2 P 8b4bb0b1671444b7b6c917963e957b6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26a4aa858ba14954b2e505694f779a86" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 45693 } } peers { permanent_uuid: "09ca737613794860ac9e00105c0b5e73" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 41037 } } peers { permanent_uuid: "8b4bb0b1671444b7b6c917963e957b6d" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 39017 } }
I20260926 15:08:57.602843 14680 ts_tablet_manager.cc:1428] T eb562d4b848d4488b3505840593c04a2 P 8b4bb0b1671444b7b6c917963e957b6d: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20260926 15:08:57.608036 14683 tablet_bootstrap.cc:492] T eb562d4b848d4488b3505840593c04a2 P 26a4aa858ba14954b2e505694f779a86: No bootstrap required, opened a new log
I20260926 15:08:57.608245 14683 ts_tablet_manager.cc:1397] T eb562d4b848d4488b3505840593c04a2 P 26a4aa858ba14954b2e505694f779a86: Time spent bootstrapping tablet: real 0.020s user 0.001s sys 0.009s
I20260926 15:08:57.609032 14731 raft_consensus.cc:2229] T 2806d4989e1d453698a68917849b10fe P 8b4bb0b1671444b7b6c917963e957b6d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 15:08:57.608990 14683 raft_consensus.cc:348] T eb562d4b848d4488b3505840593c04a2 P 26a4aa858ba14954b2e505694f779a86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26a4aa858ba14954b2e505694f779a86" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 45693 } } peers { permanent_uuid: "09ca737613794860ac9e00105c0b5e73" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 41037 } } peers { permanent_uuid: "8b4bb0b1671444b7b6c917963e957b6d" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 39017 } }
I20260926 15:08:57.609357 14731 raft_consensus.cc:2258] T 2806d4989e1d453698a68917849b10fe P 8b4bb0b1671444b7b6c917963e957b6d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:08:57.609381 14683 raft_consensus.cc:374] T eb562d4b848d4488b3505840593c04a2 P 26a4aa858ba14954b2e505694f779a86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:08:57.596843 14681 raft_consensus.cc:348] T eb562d4b848d4488b3505840593c04a2 P 09ca737613794860ac9e00105c0b5e73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26a4aa858ba14954b2e505694f779a86" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 45693 } } peers { permanent_uuid: "09ca737613794860ac9e00105c0b5e73" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 41037 } } peers { permanent_uuid: "8b4bb0b1671444b7b6c917963e957b6d" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 39017 } }
I20260926 15:08:57.609712 14681 raft_consensus.cc:374] T eb562d4b848d4488b3505840593c04a2 P 09ca737613794860ac9e00105c0b5e73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:08:57.609807 14681 raft_consensus.cc:729] T eb562d4b848d4488b3505840593c04a2 P 09ca737613794860ac9e00105c0b5e73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09ca737613794860ac9e00105c0b5e73, State: Initialized, Role: FOLLOWER
I20260926 15:08:57.609539 14683 raft_consensus.cc:729] T eb562d4b848d4488b3505840593c04a2 P 26a4aa858ba14954b2e505694f779a86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26a4aa858ba14954b2e505694f779a86, State: Initialized, Role: FOLLOWER
I20260926 15:08:57.610077 14681 consensus_queue.cc:260] T eb562d4b848d4488b3505840593c04a2 P 09ca737613794860ac9e00105c0b5e73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26a4aa858ba14954b2e505694f779a86" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 45693 } } peers { permanent_uuid: "09ca737613794860ac9e00105c0b5e73" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 41037 } } peers { permanent_uuid: "8b4bb0b1671444b7b6c917963e957b6d" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 39017 } }
I20260926 15:08:57.610195 14683 consensus_queue.cc:260] T eb562d4b848d4488b3505840593c04a2 P 26a4aa858ba14954b2e505694f779a86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26a4aa858ba14954b2e505694f779a86" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 45693 } } peers { permanent_uuid: "09ca737613794860ac9e00105c0b5e73" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 41037 } } peers { permanent_uuid: "8b4bb0b1671444b7b6c917963e957b6d" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 39017 } }
I20260926 15:08:57.610680 14681 ts_tablet_manager.cc:1428] T eb562d4b848d4488b3505840593c04a2 P 09ca737613794860ac9e00105c0b5e73: Time spent starting tablet: real 0.015s user 0.001s sys 0.001s
I20260926 15:08:57.611398 14683 ts_tablet_manager.cc:1428] T eb562d4b848d4488b3505840593c04a2 P 26a4aa858ba14954b2e505694f779a86: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260926 15:08:57.616454 14731 ts_tablet_manager.cc:1905] T 2806d4989e1d453698a68917849b10fe P 8b4bb0b1671444b7b6c917963e957b6d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:08:57.625931 14733 ts_tablet_manager.cc:1918] T 2806d4989e1d453698a68917849b10fe P 26a4aa858ba14954b2e505694f779a86: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 15:08:57.626168 14733 log.cc:1196] T 2806d4989e1d453698a68917849b10fe P 26a4aa858ba14954b2e505694f779a86: Deleting WAL directory at /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435318278681-12427-0/minicluster-data/ts-0/wal/wals/2806d4989e1d453698a68917849b10fe
I20260926 15:08:57.626775 14733 ts_tablet_manager.cc:1939] T 2806d4989e1d453698a68917849b10fe P 26a4aa858ba14954b2e505694f779a86: Deleting consensus metadata
I20260926 15:08:57.627954 14732 ts_tablet_manager.cc:1918] T 2806d4989e1d453698a68917849b10fe P 09ca737613794860ac9e00105c0b5e73: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 15:08:57.628144 14732 log.cc:1196] T 2806d4989e1d453698a68917849b10fe P 09ca737613794860ac9e00105c0b5e73: Deleting WAL directory at /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435318278681-12427-0/minicluster-data/ts-1/wal/wals/2806d4989e1d453698a68917849b10fe
I20260926 15:08:57.628688 14732 ts_tablet_manager.cc:1939] T 2806d4989e1d453698a68917849b10fe P 09ca737613794860ac9e00105c0b5e73: Deleting consensus metadata
I20260926 15:08:57.629969 14161 catalog_manager.cc:4841] TS 26a4aa858ba14954b2e505694f779a86 (127.12.34.193:45693): tablet 2806d4989e1d453698a68917849b10fe (table test-table [id=1430f2be6bc94f85991e8d3fa3a24b2d]) successfully deleted
I20260926 15:08:57.631162 14162 catalog_manager.cc:4841] TS 09ca737613794860ac9e00105c0b5e73 (127.12.34.194:41037): tablet 2806d4989e1d453698a68917849b10fe (table test-table [id=1430f2be6bc94f85991e8d3fa3a24b2d]) successfully deleted
I20260926 15:08:57.634438 14731 ts_tablet_manager.cc:1918] T 2806d4989e1d453698a68917849b10fe P 8b4bb0b1671444b7b6c917963e957b6d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 15:08:57.634688 14731 log.cc:1196] T 2806d4989e1d453698a68917849b10fe P 8b4bb0b1671444b7b6c917963e957b6d: Deleting WAL directory at /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435318278681-12427-0/minicluster-data/ts-2/wal/wals/2806d4989e1d453698a68917849b10fe
I20260926 15:08:57.635665 14731 ts_tablet_manager.cc:1939] T 2806d4989e1d453698a68917849b10fe P 8b4bb0b1671444b7b6c917963e957b6d: Deleting consensus metadata
I20260926 15:08:57.638168 14160 catalog_manager.cc:4841] TS 8b4bb0b1671444b7b6c917963e957b6d (127.12.34.195:39017): tablet 2806d4989e1d453698a68917849b10fe (table test-table [id=1430f2be6bc94f85991e8d3fa3a24b2d]) successfully deleted
I20260926 15:08:57.877913 14705 raft_consensus.cc:482] T eb562d4b848d4488b3505840593c04a2 P 8b4bb0b1671444b7b6c917963e957b6d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 15:08:57.878144 14705 raft_consensus.cc:504] T eb562d4b848d4488b3505840593c04a2 P 8b4bb0b1671444b7b6c917963e957b6d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26a4aa858ba14954b2e505694f779a86" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 45693 } } peers { permanent_uuid: "09ca737613794860ac9e00105c0b5e73" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 41037 } } peers { permanent_uuid: "8b4bb0b1671444b7b6c917963e957b6d" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 39017 } }
I20260926 15:08:57.878762 14705 leader_election.cc:288] T eb562d4b848d4488b3505840593c04a2 P 8b4bb0b1671444b7b6c917963e957b6d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 26a4aa858ba14954b2e505694f779a86 (127.12.34.193:45693), 09ca737613794860ac9e00105c0b5e73 (127.12.34.194:41037)
I20260926 15:08:57.879355 14476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb562d4b848d4488b3505840593c04a2" candidate_uuid: "8b4bb0b1671444b7b6c917963e957b6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ca737613794860ac9e00105c0b5e73" is_pre_election: true
I20260926 15:08:57.879369 14333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb562d4b848d4488b3505840593c04a2" candidate_uuid: "8b4bb0b1671444b7b6c917963e957b6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26a4aa858ba14954b2e505694f779a86" is_pre_election: true
I20260926 15:08:57.879616 14333 raft_consensus.cc:2454] T eb562d4b848d4488b3505840593c04a2 P 26a4aa858ba14954b2e505694f779a86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b4bb0b1671444b7b6c917963e957b6d in term 0.
I20260926 15:08:57.879634 14476 raft_consensus.cc:2454] T eb562d4b848d4488b3505840593c04a2 P 09ca737613794860ac9e00105c0b5e73 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b4bb0b1671444b7b6c917963e957b6d in term 0.
I20260926 15:08:57.880062 14544 leader_election.cc:302] T eb562d4b848d4488b3505840593c04a2 P 8b4bb0b1671444b7b6c917963e957b6d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ca737613794860ac9e00105c0b5e73, 8b4bb0b1671444b7b6c917963e957b6d; no voters:
I20260926 15:08:57.880365 14705 raft_consensus.cc:2781] T eb562d4b848d4488b3505840593c04a2 P 8b4bb0b1671444b7b6c917963e957b6d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 15:08:57.880505 14705 raft_consensus.cc:482] T eb562d4b848d4488b3505840593c04a2 P 8b4bb0b1671444b7b6c917963e957b6d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 15:08:57.880625 14705 raft_consensus.cc:3037] T eb562d4b848d4488b3505840593c04a2 P 8b4bb0b1671444b7b6c917963e957b6d [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:08:57.882828 14705 raft_consensus.cc:504] T eb562d4b848d4488b3505840593c04a2 P 8b4bb0b1671444b7b6c917963e957b6d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26a4aa858ba14954b2e505694f779a86" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 45693 } } peers { permanent_uuid: "09ca737613794860ac9e00105c0b5e73" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 41037 } } peers { permanent_uuid: "8b4bb0b1671444b7b6c917963e957b6d" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 39017 } }
I20260926 15:08:57.883423 14705 leader_election.cc:288] T eb562d4b848d4488b3505840593c04a2 P 8b4bb0b1671444b7b6c917963e957b6d [CANDIDATE]: Term 1 election: Requested vote from peers 26a4aa858ba14954b2e505694f779a86 (127.12.34.193:45693), 09ca737613794860ac9e00105c0b5e73 (127.12.34.194:41037)
I20260926 15:08:57.884161 14476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb562d4b848d4488b3505840593c04a2" candidate_uuid: "8b4bb0b1671444b7b6c917963e957b6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ca737613794860ac9e00105c0b5e73"
I20260926 15:08:57.884227 14333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb562d4b848d4488b3505840593c04a2" candidate_uuid: "8b4bb0b1671444b7b6c917963e957b6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26a4aa858ba14954b2e505694f779a86"
I20260926 15:08:57.884369 14476 raft_consensus.cc:3037] T eb562d4b848d4488b3505840593c04a2 P 09ca737613794860ac9e00105c0b5e73 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:08:57.884425 14333 raft_consensus.cc:3037] T eb562d4b848d4488b3505840593c04a2 P 26a4aa858ba14954b2e505694f779a86 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:08:57.887432 14333 raft_consensus.cc:2454] T eb562d4b848d4488b3505840593c04a2 P 26a4aa858ba14954b2e505694f779a86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b4bb0b1671444b7b6c917963e957b6d in term 1.
I20260926 15:08:57.887472 14476 raft_consensus.cc:2454] T eb562d4b848d4488b3505840593c04a2 P 09ca737613794860ac9e00105c0b5e73 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b4bb0b1671444b7b6c917963e957b6d in term 1.
I20260926 15:08:57.887856 14544 leader_election.cc:302] T eb562d4b848d4488b3505840593c04a2 P 8b4bb0b1671444b7b6c917963e957b6d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ca737613794860ac9e00105c0b5e73, 8b4bb0b1671444b7b6c917963e957b6d; no voters:
I20260926 15:08:57.888211 14705 raft_consensus.cc:2781] T eb562d4b848d4488b3505840593c04a2 P 8b4bb0b1671444b7b6c917963e957b6d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:08:57.888379 14705 raft_consensus.cc:686] T eb562d4b848d4488b3505840593c04a2 P 8b4bb0b1671444b7b6c917963e957b6d [term 1 LEADER]: Becoming Leader. State: Replica: 8b4bb0b1671444b7b6c917963e957b6d, State: Running, Role: LEADER
I20260926 15:08:57.888660 14705 consensus_queue.cc:237] T eb562d4b848d4488b3505840593c04a2 P 8b4bb0b1671444b7b6c917963e957b6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26a4aa858ba14954b2e505694f779a86" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 45693 } } peers { permanent_uuid: "09ca737613794860ac9e00105c0b5e73" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 41037 } } peers { permanent_uuid: "8b4bb0b1671444b7b6c917963e957b6d" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 39017 } }
I20260926 15:08:57.891261 14174 catalog_manager.cc:5495] T eb562d4b848d4488b3505840593c04a2 P 8b4bb0b1671444b7b6c917963e957b6d reported cstate change: term changed from 0 to 1, leader changed from <none> to 8b4bb0b1671444b7b6c917963e957b6d (127.12.34.195). New cstate: current_term: 1 leader_uuid: "8b4bb0b1671444b7b6c917963e957b6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26a4aa858ba14954b2e505694f779a86" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 45693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09ca737613794860ac9e00105c0b5e73" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 41037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b4bb0b1671444b7b6c917963e957b6d" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 39017 } health_report { overall_health: HEALTHY } } }
I20260926 15:08:57.975561 14675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:08:57.966553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36858 (local address 127.12.34.195:39017)
0926 15:08:57.966760 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0926 15:08:57.966766 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:08:57.969633 (+ 2867us) server_negotiation.cc:407] Connection header received
0926 15:08:57.969856 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:08:57.969861 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:08:57.969943 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:08:57.970039 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 15:08:57.970478 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:08:57.971209 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:08:57.973202 (+ 1993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:08:57.973692 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:08:57.973794 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:08:57.974710 (+ 916us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 15:08:57.974824 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 15:08:57.974895 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 15:08:57.975198 (+ 303us) server_negotiation.cc:299] Negotiation successful
0926 15:08:57.975290 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 15:08:57.988617 14675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:08:57.983168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.195:32921 (local address 127.12.34.195:39017)
0926 15:08:57.983444 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0926 15:08:57.983451 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:08:57.983527 (+ 76us) server_negotiation.cc:407] Connection header received
0926 15:08:57.983668 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:08:57.983676 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:08:57.983758 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:08:57.983859 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 15:08:57.984135 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:08:57.985320 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:08:57.986636 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:08:57.987464 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:08:57.987585 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:08:57.987907 (+ 322us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 15:08:57.987999 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 15:08:57.988085 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 15:08:57.988211 (+ 126us) server_negotiation.cc:299] Negotiation successful
0926 15:08:57.988270 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":127}
I20260926 15:08:57.988637 14727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:08:57.983202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.195:39017 (local address 127.12.34.195:32921)
0926 15:08:57.983460 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0926 15:08:57.983481 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 15:08:57.983567 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:08:57.983865 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:08:57.983868 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:08:57.983876 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 15:08:57.984026 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:08:57.984032 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:08:57.985511 (+ 1479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:08:57.985515 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:08:57.986468 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:08:57.986479 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:08:57.987605 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:08:57.987610 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:08:57.987734 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:08:57.987763 (+ 29us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 15:08:57.988085 (+ 322us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 15:08:57.988097 (+ 12us) client_negotiation.cc:769] Sending connection context
0926 15:08:57.988185 (+ 88us) client_negotiation.cc:240] Negotiation successful
0926 15:08:57.988311 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137}
I20260926 15:08:57.997339 14333 raft_consensus.cc:1261] T eb562d4b848d4488b3505840593c04a2 P 26a4aa858ba14954b2e505694f779a86 [term 1 FOLLOWER]: Refusing update from remote peer 8b4bb0b1671444b7b6c917963e957b6d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:08:57.997967 14476 raft_consensus.cc:1261] T eb562d4b848d4488b3505840593c04a2 P 09ca737613794860ac9e00105c0b5e73 [term 1 FOLLOWER]: Refusing update from remote peer 8b4bb0b1671444b7b6c917963e957b6d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:08:57.997974 14705 consensus_queue.cc:1035] T eb562d4b848d4488b3505840593c04a2 P 8b4bb0b1671444b7b6c917963e957b6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "26a4aa858ba14954b2e505694f779a86" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 45693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:08:57.999003 14705 consensus_queue.cc:1035] T eb562d4b848d4488b3505840593c04a2 P 8b4bb0b1671444b7b6c917963e957b6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "09ca737613794860ac9e00105c0b5e73" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 15:08:58.020076 14710 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260926 15:08:59.621546 14175 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46522:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 15:08:59.621778 14175 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46522:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 15:08:59.625602 14175 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 737da02a56c94512be48ba241fbb275d: Sending DeleteTablet for 3 replicas of tablet eb562d4b848d4488b3505840593c04a2
I20260926 15:08:59.626292 14312 tablet_service.cc:1514] Processing DeleteTablet for tablet eb562d4b848d4488b3505840593c04a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:08:59 UTC) from {username='kudu', principal='kudu/127.12.34.254@KRBTEST.COM'} at 127.0.0.1:42090
I20260926 15:08:59.626374 14454 tablet_service.cc:1514] Processing DeleteTablet for tablet eb562d4b848d4488b3505840593c04a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:08:59 UTC) from {username='kudu', principal='kudu/127.12.34.254@KRBTEST.COM'} at 127.0.0.1:37124
I20260926 15:08:59.626569 14590 tablet_service.cc:1514] Processing DeleteTablet for tablet eb562d4b848d4488b3505840593c04a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:08:59 UTC) from {username='kudu', principal='kudu/127.12.34.254@KRBTEST.COM'} at 127.0.0.1:36840
I20260926 15:08:59.636673 14767 tablet_replica.cc:330] T eb562d4b848d4488b3505840593c04a2 P 8b4bb0b1671444b7b6c917963e957b6d: stopping tablet replica
I20260926 15:08:59.637053 14767 raft_consensus.cc:2229] T eb562d4b848d4488b3505840593c04a2 P 8b4bb0b1671444b7b6c917963e957b6d [term 1 LEADER]: Raft consensus shutting down.
I20260926 15:08:59.637822 14767 raft_consensus.cc:2258] T eb562d4b848d4488b3505840593c04a2 P 8b4bb0b1671444b7b6c917963e957b6d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:08:59.638520 14766 tablet_replica.cc:330] T eb562d4b848d4488b3505840593c04a2 P 09ca737613794860ac9e00105c0b5e73: stopping tablet replica
I20260926 15:08:59.638738 14765 tablet_replica.cc:330] T eb562d4b848d4488b3505840593c04a2 P 26a4aa858ba14954b2e505694f779a86: stopping tablet replica
I20260926 15:08:59.638929 14765 raft_consensus.cc:2229] T eb562d4b848d4488b3505840593c04a2 P 26a4aa858ba14954b2e505694f779a86 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 15:08:59.639160 14765 raft_consensus.cc:2258] T eb562d4b848d4488b3505840593c04a2 P 26a4aa858ba14954b2e505694f779a86 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:08:59.639245 14767 ts_tablet_manager.cc:1905] T eb562d4b848d4488b3505840593c04a2 P 8b4bb0b1671444b7b6c917963e957b6d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:08:59.639524 14766 raft_consensus.cc:2229] T eb562d4b848d4488b3505840593c04a2 P 09ca737613794860ac9e00105c0b5e73 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 15:08:59.639806 14766 raft_consensus.cc:2258] T eb562d4b848d4488b3505840593c04a2 P 09ca737613794860ac9e00105c0b5e73 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:08:59.640092 14765 ts_tablet_manager.cc:1905] T eb562d4b848d4488b3505840593c04a2 P 26a4aa858ba14954b2e505694f779a86: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:08:59.641687 14766 ts_tablet_manager.cc:1905] T eb562d4b848d4488b3505840593c04a2 P 09ca737613794860ac9e00105c0b5e73: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:08:59.646771 14765 ts_tablet_manager.cc:1918] T eb562d4b848d4488b3505840593c04a2 P 26a4aa858ba14954b2e505694f779a86: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 15:08:59.646956 14765 log.cc:1196] T eb562d4b848d4488b3505840593c04a2 P 26a4aa858ba14954b2e505694f779a86: Deleting WAL directory at /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435318278681-12427-0/minicluster-data/ts-0/wal/wals/eb562d4b848d4488b3505840593c04a2
I20260926 15:08:59.647466 14765 ts_tablet_manager.cc:1939] T eb562d4b848d4488b3505840593c04a2 P 26a4aa858ba14954b2e505694f779a86: Deleting consensus metadata
I20260926 15:08:59.647955 14767 ts_tablet_manager.cc:1918] T eb562d4b848d4488b3505840593c04a2 P 8b4bb0b1671444b7b6c917963e957b6d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 15:08:59.648115 14767 log.cc:1196] T eb562d4b848d4488b3505840593c04a2 P 8b4bb0b1671444b7b6c917963e957b6d: Deleting WAL directory at /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435318278681-12427-0/minicluster-data/ts-2/wal/wals/eb562d4b848d4488b3505840593c04a2
I20260926 15:08:59.648108 14766 ts_tablet_manager.cc:1918] T eb562d4b848d4488b3505840593c04a2 P 09ca737613794860ac9e00105c0b5e73: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 15:08:59.648238 14766 log.cc:1196] T eb562d4b848d4488b3505840593c04a2 P 09ca737613794860ac9e00105c0b5e73: Deleting WAL directory at /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435318278681-12427-0/minicluster-data/ts-1/wal/wals/eb562d4b848d4488b3505840593c04a2
I20260926 15:08:59.648599 14767 ts_tablet_manager.cc:1939] T eb562d4b848d4488b3505840593c04a2 P 8b4bb0b1671444b7b6c917963e957b6d: Deleting consensus metadata
I20260926 15:08:59.648723 14766 ts_tablet_manager.cc:1939] T eb562d4b848d4488b3505840593c04a2 P 09ca737613794860ac9e00105c0b5e73: Deleting consensus metadata
I20260926 15:08:59.648908 14161 catalog_manager.cc:4841] TS 26a4aa858ba14954b2e505694f779a86 (127.12.34.193:45693): tablet eb562d4b848d4488b3505840593c04a2 (table test-table [id=db53f1b14a6e489299fc29cff7a37838]) successfully deleted
I20260926 15:08:59.649634 14162 catalog_manager.cc:4841] TS 09ca737613794860ac9e00105c0b5e73 (127.12.34.194:41037): tablet eb562d4b848d4488b3505840593c04a2 (table test-table [id=db53f1b14a6e489299fc29cff7a37838]) successfully deleted
I20260926 15:08:59.649659 14160 catalog_manager.cc:4841] TS 8b4bb0b1671444b7b6c917963e957b6d (127.12.34.195:39017): tablet eb562d4b848d4488b3505840593c04a2 (table test-table [id=db53f1b14a6e489299fc29cff7a37838]) successfully deleted
Sep 26 15:08:59 dist-test-slave-865v krb5kdc[14131](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435339, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 15:08:59 dist-test-slave-865v krb5kdc[14131](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435339, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.12.34.254@KRBTEST.COM
I20260926 15:08:59.682797 14775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:08:59.669349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38560 (local address 127.12.34.254:45313)
0926 15:08:59.669918 (+ 569us) server_negotiation.cc:206] Beginning negotiation
0926 15:08:59.669923 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:08:59.669991 (+ 68us) server_negotiation.cc:407] Connection header received
0926 15:08:59.670161 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:08:59.670165 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:08:59.670274 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:08:59.670357 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:08:59.671664 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:08:59.672585 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:08:59.673873 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:08:59.674230 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:08:59.674318 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:08:59.677984 (+ 3666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:08:59.677989 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:08:59.677992 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:08:59.678022 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:08:59.680635 (+ 2613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:08:59.681278 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:08:59.681282 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:08:59.681284 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:08:59.681395 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:08:59.681701 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:08:59.681704 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:08:59.681706 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:08:59.681933 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:08:59.682100 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0926 15:08:59.682292 (+ 192us) server_negotiation.cc:299] Negotiation successful
0926 15:08:59.682433 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":439,"thread_start_us":300,"threads_started":1}
I20260926 15:08:59.686342 14175 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38560:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 15:08:59.686856 14175 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 15:08:59.693113 14312 tablet_service.cc:1467] Processing CreateTablet for tablet f22ee0492bec457eba396f5e1d40dd6e (DEFAULT_TABLE table=test-table [id=7d6657891f074b33b7a8a9f3d0284a1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:08:59.693384 14454 tablet_service.cc:1467] Processing CreateTablet for tablet f22ee0492bec457eba396f5e1d40dd6e (DEFAULT_TABLE table=test-table [id=7d6657891f074b33b7a8a9f3d0284a1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:08:59.693542 14312 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f22ee0492bec457eba396f5e1d40dd6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:08:59.693509 14590 tablet_service.cc:1467] Processing CreateTablet for tablet f22ee0492bec457eba396f5e1d40dd6e (DEFAULT_TABLE table=test-table [id=7d6657891f074b33b7a8a9f3d0284a1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:08:59.693778 14454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f22ee0492bec457eba396f5e1d40dd6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:08:59.693892 14590 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f22ee0492bec457eba396f5e1d40dd6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:08:59.698901 14776 tablet_bootstrap.cc:492] T f22ee0492bec457eba396f5e1d40dd6e P 26a4aa858ba14954b2e505694f779a86: Bootstrap starting.
I20260926 15:08:59.700484 14777 tablet_bootstrap.cc:492] T f22ee0492bec457eba396f5e1d40dd6e P 09ca737613794860ac9e00105c0b5e73: Bootstrap starting.
I20260926 15:08:59.700524 14776 tablet_bootstrap.cc:654] T f22ee0492bec457eba396f5e1d40dd6e P 26a4aa858ba14954b2e505694f779a86: Neither blocks nor log segments found. Creating new log.
I20260926 15:08:59.701264 14778 tablet_bootstrap.cc:492] T f22ee0492bec457eba396f5e1d40dd6e P 8b4bb0b1671444b7b6c917963e957b6d: Bootstrap starting.
I20260926 15:08:59.702863 14778 tablet_bootstrap.cc:654] T f22ee0492bec457eba396f5e1d40dd6e P 8b4bb0b1671444b7b6c917963e957b6d: Neither blocks nor log segments found. Creating new log.
I20260926 15:08:59.703320 14777 tablet_bootstrap.cc:654] T f22ee0492bec457eba396f5e1d40dd6e P 09ca737613794860ac9e00105c0b5e73: Neither blocks nor log segments found. Creating new log.
I20260926 15:08:59.704883 14778 tablet_bootstrap.cc:492] T f22ee0492bec457eba396f5e1d40dd6e P 8b4bb0b1671444b7b6c917963e957b6d: No bootstrap required, opened a new log
I20260926 15:08:59.705034 14778 ts_tablet_manager.cc:1397] T f22ee0492bec457eba396f5e1d40dd6e P 8b4bb0b1671444b7b6c917963e957b6d: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260926 15:08:59.705507 14777 tablet_bootstrap.cc:492] T f22ee0492bec457eba396f5e1d40dd6e P 09ca737613794860ac9e00105c0b5e73: No bootstrap required, opened a new log
I20260926 15:08:59.705631 14777 ts_tablet_manager.cc:1397] T f22ee0492bec457eba396f5e1d40dd6e P 09ca737613794860ac9e00105c0b5e73: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 15:08:59.705664 14776 tablet_bootstrap.cc:492] T f22ee0492bec457eba396f5e1d40dd6e P 26a4aa858ba14954b2e505694f779a86: No bootstrap required, opened a new log
I20260926 15:08:59.705792 14776 ts_tablet_manager.cc:1397] T f22ee0492bec457eba396f5e1d40dd6e P 26a4aa858ba14954b2e505694f779a86: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20260926 15:08:59.705890 14778 raft_consensus.cc:348] T f22ee0492bec457eba396f5e1d40dd6e P 8b4bb0b1671444b7b6c917963e957b6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26a4aa858ba14954b2e505694f779a86" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 45693 } } peers { permanent_uuid: "09ca737613794860ac9e00105c0b5e73" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 41037 } } peers { permanent_uuid: "8b4bb0b1671444b7b6c917963e957b6d" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 39017 } }
I20260926 15:08:59.706127 14778 raft_consensus.cc:374] T f22ee0492bec457eba396f5e1d40dd6e P 8b4bb0b1671444b7b6c917963e957b6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:08:59.706221 14778 raft_consensus.cc:729] T f22ee0492bec457eba396f5e1d40dd6e P 8b4bb0b1671444b7b6c917963e957b6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b4bb0b1671444b7b6c917963e957b6d, State: Initialized, Role: FOLLOWER
I20260926 15:08:59.706493 14776 raft_consensus.cc:348] T f22ee0492bec457eba396f5e1d40dd6e P 26a4aa858ba14954b2e505694f779a86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26a4aa858ba14954b2e505694f779a86" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 45693 } } peers { permanent_uuid: "09ca737613794860ac9e00105c0b5e73" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 41037 } } peers { permanent_uuid: "8b4bb0b1671444b7b6c917963e957b6d" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 39017 } }
I20260926 15:08:59.706696 14776 raft_consensus.cc:374] T f22ee0492bec457eba396f5e1d40dd6e P 26a4aa858ba14954b2e505694f779a86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:08:59.706522 14777 raft_consensus.cc:348] T f22ee0492bec457eba396f5e1d40dd6e P 09ca737613794860ac9e00105c0b5e73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26a4aa858ba14954b2e505694f779a86" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 45693 } } peers { permanent_uuid: "09ca737613794860ac9e00105c0b5e73" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 41037 } } peers { permanent_uuid: "8b4bb0b1671444b7b6c917963e957b6d" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 39017 } }
I20260926 15:08:59.706761 14776 raft_consensus.cc:729] T f22ee0492bec457eba396f5e1d40dd6e P 26a4aa858ba14954b2e505694f779a86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26a4aa858ba14954b2e505694f779a86, State: Initialized, Role: FOLLOWER
I20260926 15:08:59.706561 14778 consensus_queue.cc:260] T f22ee0492bec457eba396f5e1d40dd6e P 8b4bb0b1671444b7b6c917963e957b6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26a4aa858ba14954b2e505694f779a86" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 45693 } } peers { permanent_uuid: "09ca737613794860ac9e00105c0b5e73" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 41037 } } peers { permanent_uuid: "8b4bb0b1671444b7b6c917963e957b6d" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 39017 } }
I20260926 15:08:59.706765 14777 raft_consensus.cc:374] T f22ee0492bec457eba396f5e1d40dd6e P 09ca737613794860ac9e00105c0b5e73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:08:59.706861 14777 raft_consensus.cc:729] T f22ee0492bec457eba396f5e1d40dd6e P 09ca737613794860ac9e00105c0b5e73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09ca737613794860ac9e00105c0b5e73, State: Initialized, Role: FOLLOWER
I20260926 15:08:59.706979 14776 consensus_queue.cc:260] T f22ee0492bec457eba396f5e1d40dd6e P 26a4aa858ba14954b2e505694f779a86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26a4aa858ba14954b2e505694f779a86" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 45693 } } peers { permanent_uuid: "09ca737613794860ac9e00105c0b5e73" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 41037 } } peers { permanent_uuid: "8b4bb0b1671444b7b6c917963e957b6d" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 39017 } }
I20260926 15:08:59.707199 14778 ts_tablet_manager.cc:1428] T f22ee0492bec457eba396f5e1d40dd6e P 8b4bb0b1671444b7b6c917963e957b6d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260926 15:08:59.707103 14777 consensus_queue.cc:260] T f22ee0492bec457eba396f5e1d40dd6e P 09ca737613794860ac9e00105c0b5e73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26a4aa858ba14954b2e505694f779a86" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 45693 } } peers { permanent_uuid: "09ca737613794860ac9e00105c0b5e73" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 41037 } } peers { permanent_uuid: "8b4bb0b1671444b7b6c917963e957b6d" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 39017 } }
I20260926 15:08:59.708622 14777 ts_tablet_manager.cc:1428] T f22ee0492bec457eba396f5e1d40dd6e P 09ca737613794860ac9e00105c0b5e73: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 15:08:59.709271 14776 ts_tablet_manager.cc:1428] T f22ee0492bec457eba396f5e1d40dd6e P 26a4aa858ba14954b2e505694f779a86: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260926 15:08:59.862215 14783 raft_consensus.cc:482] T f22ee0492bec457eba396f5e1d40dd6e P 09ca737613794860ac9e00105c0b5e73 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 15:08:59.862480 14783 raft_consensus.cc:504] T f22ee0492bec457eba396f5e1d40dd6e P 09ca737613794860ac9e00105c0b5e73 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26a4aa858ba14954b2e505694f779a86" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 45693 } } peers { permanent_uuid: "09ca737613794860ac9e00105c0b5e73" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 41037 } } peers { permanent_uuid: "8b4bb0b1671444b7b6c917963e957b6d" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 39017 } }
I20260926 15:08:59.863175 14783 leader_election.cc:288] T f22ee0492bec457eba396f5e1d40dd6e P 09ca737613794860ac9e00105c0b5e73 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 26a4aa858ba14954b2e505694f779a86 (127.12.34.193:45693), 8b4bb0b1671444b7b6c917963e957b6d (127.12.34.195:39017)
I20260926 15:08:59.863703 14332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f22ee0492bec457eba396f5e1d40dd6e" candidate_uuid: "09ca737613794860ac9e00105c0b5e73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26a4aa858ba14954b2e505694f779a86" is_pre_election: true
I20260926 15:08:59.863783 14616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f22ee0492bec457eba396f5e1d40dd6e" candidate_uuid: "09ca737613794860ac9e00105c0b5e73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b4bb0b1671444b7b6c917963e957b6d" is_pre_election: true
I20260926 15:08:59.863912 14332 raft_consensus.cc:2454] T f22ee0492bec457eba396f5e1d40dd6e P 26a4aa858ba14954b2e505694f779a86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09ca737613794860ac9e00105c0b5e73 in term 0.
I20260926 15:08:59.864071 14616 raft_consensus.cc:2454] T f22ee0492bec457eba396f5e1d40dd6e P 8b4bb0b1671444b7b6c917963e957b6d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09ca737613794860ac9e00105c0b5e73 in term 0.
I20260926 15:08:59.864363 14404 leader_election.cc:302] T f22ee0492bec457eba396f5e1d40dd6e P 09ca737613794860ac9e00105c0b5e73 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ca737613794860ac9e00105c0b5e73, 26a4aa858ba14954b2e505694f779a86; no voters:
I20260926 15:08:59.864630 14783 raft_consensus.cc:2781] T f22ee0492bec457eba396f5e1d40dd6e P 09ca737613794860ac9e00105c0b5e73 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 15:08:59.864727 14783 raft_consensus.cc:482] T f22ee0492bec457eba396f5e1d40dd6e P 09ca737613794860ac9e00105c0b5e73 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 15:08:59.864789 14783 raft_consensus.cc:3037] T f22ee0492bec457eba396f5e1d40dd6e P 09ca737613794860ac9e00105c0b5e73 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:08:59.866963 14783 raft_consensus.cc:504] T f22ee0492bec457eba396f5e1d40dd6e P 09ca737613794860ac9e00105c0b5e73 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26a4aa858ba14954b2e505694f779a86" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 45693 } } peers { permanent_uuid: "09ca737613794860ac9e00105c0b5e73" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 41037 } } peers { permanent_uuid: "8b4bb0b1671444b7b6c917963e957b6d" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 39017 } }
I20260926 15:08:59.867587 14783 leader_election.cc:288] T f22ee0492bec457eba396f5e1d40dd6e P 09ca737613794860ac9e00105c0b5e73 [CANDIDATE]: Term 1 election: Requested vote from peers 26a4aa858ba14954b2e505694f779a86 (127.12.34.193:45693), 8b4bb0b1671444b7b6c917963e957b6d (127.12.34.195:39017)
I20260926 15:08:59.868026 14332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f22ee0492bec457eba396f5e1d40dd6e" candidate_uuid: "09ca737613794860ac9e00105c0b5e73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26a4aa858ba14954b2e505694f779a86"
I20260926 15:08:59.868022 14616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f22ee0492bec457eba396f5e1d40dd6e" candidate_uuid: "09ca737613794860ac9e00105c0b5e73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b4bb0b1671444b7b6c917963e957b6d"
I20260926 15:08:59.868235 14616 raft_consensus.cc:3037] T f22ee0492bec457eba396f5e1d40dd6e P 8b4bb0b1671444b7b6c917963e957b6d [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:08:59.868234 14332 raft_consensus.cc:3037] T f22ee0492bec457eba396f5e1d40dd6e P 26a4aa858ba14954b2e505694f779a86 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:08:59.871212 14616 raft_consensus.cc:2454] T f22ee0492bec457eba396f5e1d40dd6e P 8b4bb0b1671444b7b6c917963e957b6d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09ca737613794860ac9e00105c0b5e73 in term 1.
I20260926 15:08:59.871212 14332 raft_consensus.cc:2454] T f22ee0492bec457eba396f5e1d40dd6e P 26a4aa858ba14954b2e505694f779a86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09ca737613794860ac9e00105c0b5e73 in term 1.
I20260926 15:08:59.871575 14403 leader_election.cc:302] T f22ee0492bec457eba396f5e1d40dd6e P 09ca737613794860ac9e00105c0b5e73 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ca737613794860ac9e00105c0b5e73, 8b4bb0b1671444b7b6c917963e957b6d; no voters:
I20260926 15:08:59.871848 14783 raft_consensus.cc:2781] T f22ee0492bec457eba396f5e1d40dd6e P 09ca737613794860ac9e00105c0b5e73 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:08:59.872277 14783 raft_consensus.cc:686] T f22ee0492bec457eba396f5e1d40dd6e P 09ca737613794860ac9e00105c0b5e73 [term 1 LEADER]: Becoming Leader. State: Replica: 09ca737613794860ac9e00105c0b5e73, State: Running, Role: LEADER
I20260926 15:08:59.872546 14783 consensus_queue.cc:237] T f22ee0492bec457eba396f5e1d40dd6e P 09ca737613794860ac9e00105c0b5e73 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26a4aa858ba14954b2e505694f779a86" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 45693 } } peers { permanent_uuid: "09ca737613794860ac9e00105c0b5e73" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 41037 } } peers { permanent_uuid: "8b4bb0b1671444b7b6c917963e957b6d" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 39017 } }
I20260926 15:08:59.875029 14176 catalog_manager.cc:5495] T f22ee0492bec457eba396f5e1d40dd6e P 09ca737613794860ac9e00105c0b5e73 reported cstate change: term changed from 0 to 1, leader changed from <none> to 09ca737613794860ac9e00105c0b5e73 (127.12.34.194). New cstate: current_term: 1 leader_uuid: "09ca737613794860ac9e00105c0b5e73" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26a4aa858ba14954b2e505694f779a86" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 45693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09ca737613794860ac9e00105c0b5e73" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 41037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b4bb0b1671444b7b6c917963e957b6d" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 39017 } health_report { overall_health: UNKNOWN } } }
I20260926 15:08:59.908938 14787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:08:59.903030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42460 (local address 127.12.34.194:41037)
0926 15:08:59.903795 (+ 765us) server_negotiation.cc:206] Beginning negotiation
0926 15:08:59.903800 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:08:59.903821 (+ 21us) server_negotiation.cc:407] Connection header received
0926 15:08:59.903888 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:08:59.903891 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:08:59.903960 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:08:59.904061 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 15:08:59.904612 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:08:59.905479 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:08:59.907145 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:08:59.907609 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:08:59.907721 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:08:59.908265 (+ 544us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 15:08:59.908380 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 15:08:59.908457 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 15:08:59.908616 (+ 159us) server_negotiation.cc:299] Negotiation successful
0926 15:08:59.908684 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":653,"thread_start_us":473,"threads_started":1}
I20260926 15:08:59.917376 14616 raft_consensus.cc:1261] T f22ee0492bec457eba396f5e1d40dd6e P 8b4bb0b1671444b7b6c917963e957b6d [term 1 FOLLOWER]: Refusing update from remote peer 09ca737613794860ac9e00105c0b5e73: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:08:59.917372 14332 raft_consensus.cc:1261] T f22ee0492bec457eba396f5e1d40dd6e P 26a4aa858ba14954b2e505694f779a86 [term 1 FOLLOWER]: Refusing update from remote peer 09ca737613794860ac9e00105c0b5e73: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:08:59.917897 14784 consensus_queue.cc:1035] T f22ee0492bec457eba396f5e1d40dd6e P 09ca737613794860ac9e00105c0b5e73 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26a4aa858ba14954b2e505694f779a86" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 45693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:08:59.918184 14783 consensus_queue.cc:1035] T f22ee0492bec457eba396f5e1d40dd6e P 09ca737613794860ac9e00105c0b5e73 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b4bb0b1671444b7b6c917963e957b6d" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 39017 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:08:59.925096 14749 mvcc.cc:205] Tried to move back new op lower bound from 7333623152296984576 to 7333623152120238080. Current Snapshot: MvccSnapshot[applied={T|T < 7333623152296984576}]
I20260926 15:08:59.961432 14174 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38560:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 15:08:59.961678 14174 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38560:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 15:08:59.964102 14174 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 737da02a56c94512be48ba241fbb275d: Sending DeleteTablet for 3 replicas of tablet f22ee0492bec457eba396f5e1d40dd6e
I20260926 15:08:59.964702 14312 tablet_service.cc:1514] Processing DeleteTablet for tablet f22ee0492bec457eba396f5e1d40dd6e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:08:59 UTC) from {username='kudu', principal='kudu/127.12.34.254@KRBTEST.COM'} at 127.0.0.1:42090
I20260926 15:08:59.964840 14454 tablet_service.cc:1514] Processing DeleteTablet for tablet f22ee0492bec457eba396f5e1d40dd6e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:08:59 UTC) from {username='kudu', principal='kudu/127.12.34.254@KRBTEST.COM'} at 127.0.0.1:37124
I20260926 15:08:59.964963 14765 tablet_replica.cc:330] T f22ee0492bec457eba396f5e1d40dd6e P 26a4aa858ba14954b2e505694f779a86: stopping tablet replica
I20260926 15:08:59.964936 14590 tablet_service.cc:1514] Processing DeleteTablet for tablet f22ee0492bec457eba396f5e1d40dd6e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:08:59 UTC) from {username='kudu', principal='kudu/127.12.34.254@KRBTEST.COM'} at 127.0.0.1:36840
I20260926 15:08:59.965124 14766 tablet_replica.cc:330] T f22ee0492bec457eba396f5e1d40dd6e P 09ca737613794860ac9e00105c0b5e73: stopping tablet replica
I20260926 15:08:59.965301 14766 raft_consensus.cc:2229] T f22ee0492bec457eba396f5e1d40dd6e P 09ca737613794860ac9e00105c0b5e73 [term 1 LEADER]: Raft consensus shutting down.
I20260926 15:08:59.965358 14765 raft_consensus.cc:2229] T f22ee0492bec457eba396f5e1d40dd6e P 26a4aa858ba14954b2e505694f779a86 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 15:08:59.965517 14765 raft_consensus.cc:2258] T f22ee0492bec457eba396f5e1d40dd6e P 26a4aa858ba14954b2e505694f779a86 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:08:59.965610 14767 tablet_replica.cc:330] T f22ee0492bec457eba396f5e1d40dd6e P 8b4bb0b1671444b7b6c917963e957b6d: stopping tablet replica
I20260926 15:08:59.965620 14766 raft_consensus.cc:2258] T f22ee0492bec457eba396f5e1d40dd6e P 09ca737613794860ac9e00105c0b5e73 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:08:59.965821 14767 raft_consensus.cc:2229] T f22ee0492bec457eba396f5e1d40dd6e P 8b4bb0b1671444b7b6c917963e957b6d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 15:08:59.965994 14767 raft_consensus.cc:2258] T f22ee0492bec457eba396f5e1d40dd6e P 8b4bb0b1671444b7b6c917963e957b6d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:08:59.966050 14174 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38560:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 15:08:59.966724 14765 ts_tablet_manager.cc:1905] T f22ee0492bec457eba396f5e1d40dd6e P 26a4aa858ba14954b2e505694f779a86: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260926 15:08:59.967049 14174 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 15:08:59.967484 14766 ts_tablet_manager.cc:1905] T f22ee0492bec457eba396f5e1d40dd6e P 09ca737613794860ac9e00105c0b5e73: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:08:59.973136 14766 ts_tablet_manager.cc:1918] T f22ee0492bec457eba396f5e1d40dd6e P 09ca737613794860ac9e00105c0b5e73: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 15:08:59.973315 14766 log.cc:1196] T f22ee0492bec457eba396f5e1d40dd6e P 09ca737613794860ac9e00105c0b5e73: Deleting WAL directory at /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435318278681-12427-0/minicluster-data/ts-1/wal/wals/f22ee0492bec457eba396f5e1d40dd6e
I20260926 15:08:59.973671 14765 ts_tablet_manager.cc:1918] T f22ee0492bec457eba396f5e1d40dd6e P 26a4aa858ba14954b2e505694f779a86: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 15:08:59.973832 14765 log.cc:1196] T f22ee0492bec457eba396f5e1d40dd6e P 26a4aa858ba14954b2e505694f779a86: Deleting WAL directory at /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435318278681-12427-0/minicluster-data/ts-0/wal/wals/f22ee0492bec457eba396f5e1d40dd6e
I20260926 15:08:59.974052 14766 ts_tablet_manager.cc:1939] T f22ee0492bec457eba396f5e1d40dd6e P 09ca737613794860ac9e00105c0b5e73: Deleting consensus metadata
I20260926 15:08:59.974004 14312 tablet_service.cc:1467] Processing CreateTablet for tablet 642ac592ac0e49458f98fe2040ed8c1b (DEFAULT_TABLE table=test-table [id=cc4ac9159a15463093e9179f6e553bf9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:08:59.974390 14312 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 642ac592ac0e49458f98fe2040ed8c1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:08:59.974314 14454 tablet_service.cc:1467] Processing CreateTablet for tablet 642ac592ac0e49458f98fe2040ed8c1b (DEFAULT_TABLE table=test-table [id=cc4ac9159a15463093e9179f6e553bf9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:08:59.974700 14454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 642ac592ac0e49458f98fe2040ed8c1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:08:59.974752 14590 tablet_service.cc:1467] Processing CreateTablet for tablet 642ac592ac0e49458f98fe2040ed8c1b (DEFAULT_TABLE table=test-table [id=cc4ac9159a15463093e9179f6e553bf9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:08:59.974992 14162 catalog_manager.cc:4841] TS 09ca737613794860ac9e00105c0b5e73 (127.12.34.194:41037): tablet f22ee0492bec457eba396f5e1d40dd6e (table test-table [id=7d6657891f074b33b7a8a9f3d0284a1e]) successfully deleted
I20260926 15:08:59.975128 14590 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 642ac592ac0e49458f98fe2040ed8c1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:08:59.975980 14767 ts_tablet_manager.cc:1905] T f22ee0492bec457eba396f5e1d40dd6e P 8b4bb0b1671444b7b6c917963e957b6d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:08:59.980415 14776 tablet_bootstrap.cc:492] T 642ac592ac0e49458f98fe2040ed8c1b P 26a4aa858ba14954b2e505694f779a86: Bootstrap starting.
I20260926 15:08:59.980954 14777 tablet_bootstrap.cc:492] T 642ac592ac0e49458f98fe2040ed8c1b P 09ca737613794860ac9e00105c0b5e73: Bootstrap starting.
I20260926 15:08:59.981498 14765 ts_tablet_manager.cc:1939] T f22ee0492bec457eba396f5e1d40dd6e P 26a4aa858ba14954b2e505694f779a86: Deleting consensus metadata
I20260926 15:08:59.982016 14776 tablet_bootstrap.cc:654] T 642ac592ac0e49458f98fe2040ed8c1b P 26a4aa858ba14954b2e505694f779a86: Neither blocks nor log segments found. Creating new log.
I20260926 15:08:59.982503 14777 tablet_bootstrap.cc:654] T 642ac592ac0e49458f98fe2040ed8c1b P 09ca737613794860ac9e00105c0b5e73: Neither blocks nor log segments found. Creating new log.
I20260926 15:08:59.982473 14161 catalog_manager.cc:4841] TS 26a4aa858ba14954b2e505694f779a86 (127.12.34.193:45693): tablet f22ee0492bec457eba396f5e1d40dd6e (table test-table [id=7d6657891f074b33b7a8a9f3d0284a1e]) successfully deleted
I20260926 15:08:59.983052 14767 ts_tablet_manager.cc:1918] T f22ee0492bec457eba396f5e1d40dd6e P 8b4bb0b1671444b7b6c917963e957b6d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 15:08:59.983189 14767 log.cc:1196] T f22ee0492bec457eba396f5e1d40dd6e P 8b4bb0b1671444b7b6c917963e957b6d: Deleting WAL directory at /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435318278681-12427-0/minicluster-data/ts-2/wal/wals/f22ee0492bec457eba396f5e1d40dd6e
I20260926 15:08:59.983589 14767 ts_tablet_manager.cc:1939] T f22ee0492bec457eba396f5e1d40dd6e P 8b4bb0b1671444b7b6c917963e957b6d: Deleting consensus metadata
I20260926 15:08:59.984015 14776 tablet_bootstrap.cc:492] T 642ac592ac0e49458f98fe2040ed8c1b P 26a4aa858ba14954b2e505694f779a86: No bootstrap required, opened a new log
I20260926 15:08:59.984164 14776 ts_tablet_manager.cc:1397] T 642ac592ac0e49458f98fe2040ed8c1b P 26a4aa858ba14954b2e505694f779a86: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260926 15:08:59.984537 14160 catalog_manager.cc:4841] TS 8b4bb0b1671444b7b6c917963e957b6d (127.12.34.195:39017): tablet f22ee0492bec457eba396f5e1d40dd6e (table test-table [id=7d6657891f074b33b7a8a9f3d0284a1e]) successfully deleted
I20260926 15:08:59.984887 14776 raft_consensus.cc:348] T 642ac592ac0e49458f98fe2040ed8c1b P 26a4aa858ba14954b2e505694f779a86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26a4aa858ba14954b2e505694f779a86" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 45693 } } peers { permanent_uuid: "09ca737613794860ac9e00105c0b5e73" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 41037 } } peers { permanent_uuid: "8b4bb0b1671444b7b6c917963e957b6d" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 39017 } }
I20260926 15:08:59.985162 14776 raft_consensus.cc:374] T 642ac592ac0e49458f98fe2040ed8c1b P 26a4aa858ba14954b2e505694f779a86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:08:59.985244 14776 raft_consensus.cc:729] T 642ac592ac0e49458f98fe2040ed8c1b P 26a4aa858ba14954b2e505694f779a86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26a4aa858ba14954b2e505694f779a86, State: Initialized, Role: FOLLOWER
I20260926 15:08:59.985473 14776 consensus_queue.cc:260] T 642ac592ac0e49458f98fe2040ed8c1b P 26a4aa858ba14954b2e505694f779a86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26a4aa858ba14954b2e505694f779a86" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 45693 } } peers { permanent_uuid: "09ca737613794860ac9e00105c0b5e73" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 41037 } } peers { permanent_uuid: "8b4bb0b1671444b7b6c917963e957b6d" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 39017 } }
I20260926 15:08:59.985790 14777 tablet_bootstrap.cc:492] T 642ac592ac0e49458f98fe2040ed8c1b P 09ca737613794860ac9e00105c0b5e73: No bootstrap required, opened a new log
I20260926 15:08:59.985916 14777 ts_tablet_manager.cc:1397] T 642ac592ac0e49458f98fe2040ed8c1b P 09ca737613794860ac9e00105c0b5e73: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 15:08:59.986536 14776 ts_tablet_manager.cc:1428] T 642ac592ac0e49458f98fe2040ed8c1b P 26a4aa858ba14954b2e505694f779a86: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260926 15:08:59.986562 14777 raft_consensus.cc:348] T 642ac592ac0e49458f98fe2040ed8c1b P 09ca737613794860ac9e00105c0b5e73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26a4aa858ba14954b2e505694f779a86" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 45693 } } peers { permanent_uuid: "09ca737613794860ac9e00105c0b5e73" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 41037 } } peers { permanent_uuid: "8b4bb0b1671444b7b6c917963e957b6d" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 39017 } }
I20260926 15:08:59.986773 14777 raft_consensus.cc:374] T 642ac592ac0e49458f98fe2040ed8c1b P 09ca737613794860ac9e00105c0b5e73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:08:59.986840 14777 raft_consensus.cc:729] T 642ac592ac0e49458f98fe2040ed8c1b P 09ca737613794860ac9e00105c0b5e73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09ca737613794860ac9e00105c0b5e73, State: Initialized, Role: FOLLOWER
I20260926 15:08:59.987067 14777 consensus_queue.cc:260] T 642ac592ac0e49458f98fe2040ed8c1b P 09ca737613794860ac9e00105c0b5e73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26a4aa858ba14954b2e505694f779a86" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 45693 } } peers { permanent_uuid: "09ca737613794860ac9e00105c0b5e73" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 41037 } } peers { permanent_uuid: "8b4bb0b1671444b7b6c917963e957b6d" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 39017 } }
I20260926 15:08:59.987579 14777 ts_tablet_manager.cc:1428] T 642ac592ac0e49458f98fe2040ed8c1b P 09ca737613794860ac9e00105c0b5e73: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260926 15:08:59.989033 14778 tablet_bootstrap.cc:492] T 642ac592ac0e49458f98fe2040ed8c1b P 8b4bb0b1671444b7b6c917963e957b6d: Bootstrap starting.
I20260926 15:08:59.990264 14778 tablet_bootstrap.cc:654] T 642ac592ac0e49458f98fe2040ed8c1b P 8b4bb0b1671444b7b6c917963e957b6d: Neither blocks nor log segments found. Creating new log.
I20260926 15:08:59.992519 14778 tablet_bootstrap.cc:492] T 642ac592ac0e49458f98fe2040ed8c1b P 8b4bb0b1671444b7b6c917963e957b6d: No bootstrap required, opened a new log
I20260926 15:08:59.992684 14778 ts_tablet_manager.cc:1397] T 642ac592ac0e49458f98fe2040ed8c1b P 8b4bb0b1671444b7b6c917963e957b6d: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260926 15:08:59.993307 14778 raft_consensus.cc:348] T 642ac592ac0e49458f98fe2040ed8c1b P 8b4bb0b1671444b7b6c917963e957b6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26a4aa858ba14954b2e505694f779a86" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 45693 } } peers { permanent_uuid: "09ca737613794860ac9e00105c0b5e73" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 41037 } } peers { permanent_uuid: "8b4bb0b1671444b7b6c917963e957b6d" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 39017 } }
I20260926 15:08:59.993494 14778 raft_consensus.cc:374] T 642ac592ac0e49458f98fe2040ed8c1b P 8b4bb0b1671444b7b6c917963e957b6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:08:59.993583 14778 raft_consensus.cc:729] T 642ac592ac0e49458f98fe2040ed8c1b P 8b4bb0b1671444b7b6c917963e957b6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b4bb0b1671444b7b6c917963e957b6d, State: Initialized, Role: FOLLOWER
I20260926 15:08:59.993803 14778 consensus_queue.cc:260] T 642ac592ac0e49458f98fe2040ed8c1b P 8b4bb0b1671444b7b6c917963e957b6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26a4aa858ba14954b2e505694f779a86" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 45693 } } peers { permanent_uuid: "09ca737613794860ac9e00105c0b5e73" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 41037 } } peers { permanent_uuid: "8b4bb0b1671444b7b6c917963e957b6d" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 39017 } }
I20260926 15:08:59.994289 14778 ts_tablet_manager.cc:1428] T 642ac592ac0e49458f98fe2040ed8c1b P 8b4bb0b1671444b7b6c917963e957b6d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260926 15:09:00.189522 14782 raft_consensus.cc:482] T 642ac592ac0e49458f98fe2040ed8c1b P 26a4aa858ba14954b2e505694f779a86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 15:09:00.189816 14782 raft_consensus.cc:504] T 642ac592ac0e49458f98fe2040ed8c1b P 26a4aa858ba14954b2e505694f779a86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26a4aa858ba14954b2e505694f779a86" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 45693 } } peers { permanent_uuid: "09ca737613794860ac9e00105c0b5e73" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 41037 } } peers { permanent_uuid: "8b4bb0b1671444b7b6c917963e957b6d" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 39017 } }
I20260926 15:09:00.191406 14782 leader_election.cc:288] T 642ac592ac0e49458f98fe2040ed8c1b P 26a4aa858ba14954b2e505694f779a86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09ca737613794860ac9e00105c0b5e73 (127.12.34.194:41037), 8b4bb0b1671444b7b6c917963e957b6d (127.12.34.195:39017)
I20260926 15:09:00.198366 14787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:00.191779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.193:55817 (local address 127.12.34.194:41037)
0926 15:09:00.191985 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:00.191994 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:00.192615 (+ 621us) server_negotiation.cc:407] Connection header received
0926 15:09:00.192815 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:00.192822 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:00.192901 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:00.192999 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:09:00.193697 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:00.194431 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:00.196249 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:00.197354 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:00.197492 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:00.197560 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 15:09:00.197941 (+ 381us) server_negotiation.cc:299] Negotiation successful
0926 15:09:00.198082 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 15:09:00.198412 14796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:00.191627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.194:41037 (local address 127.12.34.193:55817)
0926 15:09:00.192456 (+ 829us) negotiation.cc:107] Waiting for socket to connect
0926 15:09:00.192478 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 15:09:00.192669 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:09:00.193031 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:09:00.193036 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:09:00.193098 (+ 62us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:09:00.193521 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:00.193535 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:00.194570 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:00.194576 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:00.196084 (+ 1508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:00.196096 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:00.197553 (+ 1457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:00.197562 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:00.197752 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:00.197791 (+ 39us) client_negotiation.cc:769] Sending connection context
0926 15:09:00.197900 (+ 109us) client_negotiation.cc:240] Negotiation successful
0926 15:09:00.198043 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":634,"mutex_wait_us":1,"thread_start_us":413,"threads_started":1}
I20260926 15:09:00.198988 14798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:00.191880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.195:39017 (local address 127.12.34.193:42089)
0926 15:09:00.192710 (+ 830us) negotiation.cc:107] Waiting for socket to connect
0926 15:09:00.192740 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0926 15:09:00.192863 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:09:00.193278 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:09:00.193281 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:09:00.193301 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:09:00.193662 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:00.193675 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:00.194823 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:00.194828 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:00.196645 (+ 1817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:00.196662 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:00.198024 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:00.198032 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:00.198238 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:00.198276 (+ 38us) client_negotiation.cc:769] Sending connection context
0926 15:09:00.198383 (+ 107us) client_negotiation.cc:240] Negotiation successful
0926 15:09:00.198655 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":711,"thread_start_us":487,"threads_started":1}
I20260926 15:09:00.199075 14476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "642ac592ac0e49458f98fe2040ed8c1b" candidate_uuid: "26a4aa858ba14954b2e505694f779a86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ca737613794860ac9e00105c0b5e73" is_pre_election: true
I20260926 15:09:00.199323 14797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:00.191878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.193:42089 (local address 127.12.34.195:39017)
0926 15:09:00.192410 (+ 532us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:00.192419 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:00.192829 (+ 410us) server_negotiation.cc:407] Connection header received
0926 15:09:00.193036 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:00.193041 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:00.193142 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:00.193248 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:09:00.193822 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:00.194664 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:00.196812 (+ 2148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:00.197816 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:00.197984 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:00.198064 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 15:09:00.198823 (+ 759us) server_negotiation.cc:299] Negotiation successful
0926 15:09:00.199000 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":397,"thread_start_us":315,"threads_started":1}
I20260926 15:09:00.199322 14476 raft_consensus.cc:2454] T 642ac592ac0e49458f98fe2040ed8c1b P 09ca737613794860ac9e00105c0b5e73 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26a4aa858ba14954b2e505694f779a86 in term 0.
I20260926 15:09:00.199856 14265 leader_election.cc:302] T 642ac592ac0e49458f98fe2040ed8c1b P 26a4aa858ba14954b2e505694f779a86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ca737613794860ac9e00105c0b5e73, 26a4aa858ba14954b2e505694f779a86; no voters:
I20260926 15:09:00.200382 14616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "642ac592ac0e49458f98fe2040ed8c1b" candidate_uuid: "26a4aa858ba14954b2e505694f779a86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b4bb0b1671444b7b6c917963e957b6d" is_pre_election: true
I20260926 15:09:00.200603 14616 raft_consensus.cc:2454] T 642ac592ac0e49458f98fe2040ed8c1b P 8b4bb0b1671444b7b6c917963e957b6d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26a4aa858ba14954b2e505694f779a86 in term 0.
I20260926 15:09:00.200683 14782 raft_consensus.cc:2781] T 642ac592ac0e49458f98fe2040ed8c1b P 26a4aa858ba14954b2e505694f779a86 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 15:09:00.200839 14782 raft_consensus.cc:482] T 642ac592ac0e49458f98fe2040ed8c1b P 26a4aa858ba14954b2e505694f779a86 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 15:09:00.200950 14782 raft_consensus.cc:3037] T 642ac592ac0e49458f98fe2040ed8c1b P 26a4aa858ba14954b2e505694f779a86 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:09:00.203138 14782 raft_consensus.cc:504] T 642ac592ac0e49458f98fe2040ed8c1b P 26a4aa858ba14954b2e505694f779a86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26a4aa858ba14954b2e505694f779a86" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 45693 } } peers { permanent_uuid: "09ca737613794860ac9e00105c0b5e73" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 41037 } } peers { permanent_uuid: "8b4bb0b1671444b7b6c917963e957b6d" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 39017 } }
I20260926 15:09:00.203766 14782 leader_election.cc:288] T 642ac592ac0e49458f98fe2040ed8c1b P 26a4aa858ba14954b2e505694f779a86 [CANDIDATE]: Term 1 election: Requested vote from peers 09ca737613794860ac9e00105c0b5e73 (127.12.34.194:41037), 8b4bb0b1671444b7b6c917963e957b6d (127.12.34.195:39017)
I20260926 15:09:00.204226 14476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "642ac592ac0e49458f98fe2040ed8c1b" candidate_uuid: "26a4aa858ba14954b2e505694f779a86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ca737613794860ac9e00105c0b5e73"
I20260926 15:09:00.204447 14476 raft_consensus.cc:3037] T 642ac592ac0e49458f98fe2040ed8c1b P 09ca737613794860ac9e00105c0b5e73 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:09:00.204533 14616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "642ac592ac0e49458f98fe2040ed8c1b" candidate_uuid: "26a4aa858ba14954b2e505694f779a86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b4bb0b1671444b7b6c917963e957b6d"
I20260926 15:09:00.204901 14616 raft_consensus.cc:3037] T 642ac592ac0e49458f98fe2040ed8c1b P 8b4bb0b1671444b7b6c917963e957b6d [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:09:00.207087 14476 raft_consensus.cc:2454] T 642ac592ac0e49458f98fe2040ed8c1b P 09ca737613794860ac9e00105c0b5e73 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26a4aa858ba14954b2e505694f779a86 in term 1.
I20260926 15:09:00.207657 14616 raft_consensus.cc:2454] T 642ac592ac0e49458f98fe2040ed8c1b P 8b4bb0b1671444b7b6c917963e957b6d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26a4aa858ba14954b2e505694f779a86 in term 1.
I20260926 15:09:00.207862 14265 leader_election.cc:302] T 642ac592ac0e49458f98fe2040ed8c1b P 26a4aa858ba14954b2e505694f779a86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ca737613794860ac9e00105c0b5e73, 26a4aa858ba14954b2e505694f779a86; no voters:
I20260926 15:09:00.208179 14782 raft_consensus.cc:2781] T 642ac592ac0e49458f98fe2040ed8c1b P 26a4aa858ba14954b2e505694f779a86 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:09:00.208603 14782 raft_consensus.cc:686] T 642ac592ac0e49458f98fe2040ed8c1b P 26a4aa858ba14954b2e505694f779a86 [term 1 LEADER]: Becoming Leader. State: Replica: 26a4aa858ba14954b2e505694f779a86, State: Running, Role: LEADER
I20260926 15:09:00.209026 14782 consensus_queue.cc:237] T 642ac592ac0e49458f98fe2040ed8c1b P 26a4aa858ba14954b2e505694f779a86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26a4aa858ba14954b2e505694f779a86" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 45693 } } peers { permanent_uuid: "09ca737613794860ac9e00105c0b5e73" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 41037 } } peers { permanent_uuid: "8b4bb0b1671444b7b6c917963e957b6d" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 39017 } }
I20260926 15:09:00.212432 14174 catalog_manager.cc:5495] T 642ac592ac0e49458f98fe2040ed8c1b P 26a4aa858ba14954b2e505694f779a86 reported cstate change: term changed from 0 to 1, leader changed from <none> to 26a4aa858ba14954b2e505694f779a86 (127.12.34.193). New cstate: current_term: 1 leader_uuid: "26a4aa858ba14954b2e505694f779a86" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26a4aa858ba14954b2e505694f779a86" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 45693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09ca737613794860ac9e00105c0b5e73" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 41037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b4bb0b1671444b7b6c917963e957b6d" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 39017 } health_report { overall_health: UNKNOWN } } }
I20260926 15:09:00.240139 14802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:00.233893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59222 (local address 127.12.34.193:45693)
0926 15:09:00.234594 (+ 701us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:00.234598 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:00.234616 (+ 18us) server_negotiation.cc:407] Connection header received
0926 15:09:00.234691 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:00.234695 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:00.234768 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:00.234946 (+ 178us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 15:09:00.235322 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:00.236522 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:00.237842 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:00.238242 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:00.238371 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:00.238798 (+ 427us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 15:09:00.239363 (+ 565us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 15:09:00.239514 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0926 15:09:00.239603 (+ 89us) server_negotiation.cc:299] Negotiation successful
0926 15:09:00.239677 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":603,"thread_start_us":462,"threads_started":1}
I20260926 15:09:00.258276 14797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:00.252506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.193:46579 (local address 127.12.34.195:39017)
0926 15:09:00.252742 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:00.252748 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:00.253143 (+ 395us) server_negotiation.cc:407] Connection header received
0926 15:09:00.253320 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:00.253325 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:00.253392 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:00.253490 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 15:09:00.253901 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:00.254606 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:00.256541 (+ 1935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:00.256964 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:00.257062 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:00.257685 (+ 623us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 15:09:00.257790 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 15:09:00.257867 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 15:09:00.258002 (+ 135us) server_negotiation.cc:299] Negotiation successful
0926 15:09:00.258073 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 15:09:00.258426 14805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:00.252389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.195:39017 (local address 127.12.34.193:46579)
0926 15:09:00.252994 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0926 15:09:00.253010 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 15:09:00.253179 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:09:00.253496 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:09:00.253499 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:09:00.253508 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 15:09:00.253764 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:00.253771 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:00.254756 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:00.254761 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:00.256370 (+ 1609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:00.256408 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:00.257257 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:00.257262 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:00.257444 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:00.257500 (+ 56us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 15:09:00.257890 (+ 390us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 15:09:00.257900 (+ 10us) client_negotiation.cc:769] Sending connection context
0926 15:09:00.257974 (+ 74us) client_negotiation.cc:240] Negotiation successful
0926 15:09:00.258075 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":317,"threads_started":1}
I20260926 15:09:00.268573 14476 raft_consensus.cc:1261] T 642ac592ac0e49458f98fe2040ed8c1b P 09ca737613794860ac9e00105c0b5e73 [term 1 FOLLOWER]: Refusing update from remote peer 26a4aa858ba14954b2e505694f779a86: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:09:00.268692 14616 raft_consensus.cc:1261] T 642ac592ac0e49458f98fe2040ed8c1b P 8b4bb0b1671444b7b6c917963e957b6d [term 1 FOLLOWER]: Refusing update from remote peer 26a4aa858ba14954b2e505694f779a86: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:09:00.269399 14799 consensus_queue.cc:1035] T 642ac592ac0e49458f98fe2040ed8c1b P 26a4aa858ba14954b2e505694f779a86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09ca737613794860ac9e00105c0b5e73" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:09:00.269848 14782 consensus_queue.cc:1035] T 642ac592ac0e49458f98fe2040ed8c1b P 26a4aa858ba14954b2e505694f779a86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b4bb0b1671444b7b6c917963e957b6d" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 39017 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:09:00.298666 14802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:00.293474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.195:41237 (local address 127.12.34.193:45693)
0926 15:09:00.293672 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:00.293682 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:00.294102 (+ 420us) server_negotiation.cc:407] Connection header received
0926 15:09:00.294215 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:00.294219 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:00.294283 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:00.294361 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 15:09:00.294755 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:00.295612 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:00.297142 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:00.297518 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:00.297651 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:00.298009 (+ 358us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 15:09:00.298099 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 15:09:00.298192 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 15:09:00.298315 (+ 123us) server_negotiation.cc:299] Negotiation successful
0926 15:09:00.298384 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 15:09:00.298786 14811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:00.293400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.193:45693 (local address 127.12.34.195:41237)
0926 15:09:00.294011 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0926 15:09:00.294027 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 15:09:00.294130 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:09:00.294404 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:09:00.294407 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:09:00.294416 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 15:09:00.294618 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:00.294625 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:00.295755 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:00.295763 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:00.296973 (+ 1210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:00.296986 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:00.297684 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:00.297690 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:00.297824 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:00.297856 (+ 32us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 15:09:00.298193 (+ 337us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 15:09:00.298206 (+ 13us) client_negotiation.cc:769] Sending connection context
0926 15:09:00.298296 (+ 90us) client_negotiation.cc:240] Negotiation successful
0926 15:09:00.298421 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":344,"threads_started":1}
W20260926 15:09:00.299726 14749 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260926 15:09:01.914535 14174 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38560:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 15:09:01.914739 14174 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38560:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 15:09:01.918790 14174 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 737da02a56c94512be48ba241fbb275d: Sending DeleteTablet for 3 replicas of tablet 642ac592ac0e49458f98fe2040ed8c1b
I20260926 15:09:01.919467 14312 tablet_service.cc:1514] Processing DeleteTablet for tablet 642ac592ac0e49458f98fe2040ed8c1b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:09:01 UTC) from {username='kudu', principal='kudu/127.12.34.254@KRBTEST.COM'} at 127.0.0.1:42090
I20260926 15:09:01.919565 14454 tablet_service.cc:1514] Processing DeleteTablet for tablet 642ac592ac0e49458f98fe2040ed8c1b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:09:01 UTC) from {username='kudu', principal='kudu/127.12.34.254@KRBTEST.COM'} at 127.0.0.1:37124
I20260926 15:09:01.919675 14590 tablet_service.cc:1514] Processing DeleteTablet for tablet 642ac592ac0e49458f98fe2040ed8c1b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:09:01 UTC) from {username='kudu', principal='kudu/127.12.34.254@KRBTEST.COM'} at 127.0.0.1:36840
I20260926 15:09:01.920707 12427 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 14247
I20260926 15:09:01.931985 14836 tablet_replica.cc:330] T 642ac592ac0e49458f98fe2040ed8c1b P 09ca737613794860ac9e00105c0b5e73: stopping tablet replica
I20260926 15:09:01.932070 14835 tablet_replica.cc:330] T 642ac592ac0e49458f98fe2040ed8c1b P 26a4aa858ba14954b2e505694f779a86: stopping tablet replica
I20260926 15:09:01.932293 14836 raft_consensus.cc:2229] T 642ac592ac0e49458f98fe2040ed8c1b P 09ca737613794860ac9e00105c0b5e73 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 15:09:01.932367 14835 raft_consensus.cc:2229] T 642ac592ac0e49458f98fe2040ed8c1b P 26a4aa858ba14954b2e505694f779a86 [term 1 LEADER]: Raft consensus shutting down.
I20260926 15:09:01.932608 14836 raft_consensus.cc:2258] T 642ac592ac0e49458f98fe2040ed8c1b P 09ca737613794860ac9e00105c0b5e73 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:09:01.932960 14835 raft_consensus.cc:2258] T 642ac592ac0e49458f98fe2040ed8c1b P 26a4aa858ba14954b2e505694f779a86 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:09:01.933702 14836 ts_tablet_manager.cc:1905] T 642ac592ac0e49458f98fe2040ed8c1b P 09ca737613794860ac9e00105c0b5e73: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:09:01.933976 14835 ts_tablet_manager.cc:1905] T 642ac592ac0e49458f98fe2040ed8c1b P 26a4aa858ba14954b2e505694f779a86: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:09:01.940404 14836 ts_tablet_manager.cc:1918] T 642ac592ac0e49458f98fe2040ed8c1b P 09ca737613794860ac9e00105c0b5e73: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 15:09:01.940434 14835 ts_tablet_manager.cc:1918] T 642ac592ac0e49458f98fe2040ed8c1b P 26a4aa858ba14954b2e505694f779a86: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 15:09:01.940542 14835 log.cc:1196] T 642ac592ac0e49458f98fe2040ed8c1b P 26a4aa858ba14954b2e505694f779a86: Deleting WAL directory at /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435318278681-12427-0/minicluster-data/ts-0/wal/wals/642ac592ac0e49458f98fe2040ed8c1b
I20260926 15:09:01.940645 14836 log.cc:1196] T 642ac592ac0e49458f98fe2040ed8c1b P 09ca737613794860ac9e00105c0b5e73: Deleting WAL directory at /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435318278681-12427-0/minicluster-data/ts-1/wal/wals/642ac592ac0e49458f98fe2040ed8c1b
I20260926 15:09:01.941108 14835 ts_tablet_manager.cc:1939] T 642ac592ac0e49458f98fe2040ed8c1b P 26a4aa858ba14954b2e505694f779a86: Deleting consensus metadata
I20260926 15:09:01.941149 14836 ts_tablet_manager.cc:1939] T 642ac592ac0e49458f98fe2040ed8c1b P 09ca737613794860ac9e00105c0b5e73: Deleting consensus metadata
I20260926 15:09:01.942282 14162 catalog_manager.cc:4841] TS 09ca737613794860ac9e00105c0b5e73 (127.12.34.194:41037): tablet 642ac592ac0e49458f98fe2040ed8c1b (table test-table [id=cc4ac9159a15463093e9179f6e553bf9]) successfully deleted
I20260926 15:09:01.942735 14837 tablet_replica.cc:330] T 642ac592ac0e49458f98fe2040ed8c1b P 8b4bb0b1671444b7b6c917963e957b6d: stopping tablet replica
I20260926 15:09:01.943069 14837 raft_consensus.cc:2229] T 642ac592ac0e49458f98fe2040ed8c1b P 8b4bb0b1671444b7b6c917963e957b6d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 15:09:01.943416 14837 raft_consensus.cc:2258] T 642ac592ac0e49458f98fe2040ed8c1b P 8b4bb0b1671444b7b6c917963e957b6d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:09:01.943737 14161 catalog_manager.cc:4841] TS 26a4aa858ba14954b2e505694f779a86 (127.12.34.193:45693): tablet 642ac592ac0e49458f98fe2040ed8c1b (table test-table [id=cc4ac9159a15463093e9179f6e553bf9]) successfully deleted
I20260926 15:09:01.944559 14837 ts_tablet_manager.cc:1905] T 642ac592ac0e49458f98fe2040ed8c1b P 8b4bb0b1671444b7b6c917963e957b6d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:09:01.949288 14837 ts_tablet_manager.cc:1918] T 642ac592ac0e49458f98fe2040ed8c1b P 8b4bb0b1671444b7b6c917963e957b6d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 15:09:01.949420 14837 log.cc:1196] T 642ac592ac0e49458f98fe2040ed8c1b P 8b4bb0b1671444b7b6c917963e957b6d: Deleting WAL directory at /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790435318278681-12427-0/minicluster-data/ts-2/wal/wals/642ac592ac0e49458f98fe2040ed8c1b
I20260926 15:09:01.949901 14837 ts_tablet_manager.cc:1939] T 642ac592ac0e49458f98fe2040ed8c1b P 8b4bb0b1671444b7b6c917963e957b6d: Deleting consensus metadata
I20260926 15:09:01.950855 14160 catalog_manager.cc:4841] TS 8b4bb0b1671444b7b6c917963e957b6d (127.12.34.195:39017): tablet 642ac592ac0e49458f98fe2040ed8c1b (table test-table [id=cc4ac9159a15463093e9179f6e553bf9]) successfully deleted
Sep 26 15:09:01 dist-test-slave-865v krb5kdc[14131](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435339, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.12.34.193@KRBTEST.COM
I20260926 15:09:01.959884 14839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:01.921967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59232 (local address 127.12.34.193:45693)
0926 15:09:01.943738 (+ 21771us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:01.943744 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:01.944456 (+ 712us) server_negotiation.cc:407] Connection header received
0926 15:09:01.944605 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:01.944610 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:01.944691 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:01.944807 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:09:01.946312 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:01.947408 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:01.949196 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:01.949982 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:01.950128 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:01.954417 (+ 4289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:09:01.954467 (+ 50us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:09:01.954482 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:09:01.954513 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:09:01.957322 (+ 2809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:01.957934 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:01.957941 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:01.957944 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:01.958067 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:01.958423 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:01.958432 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:01.958434 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:01.958761 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:09:01.959055 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0926 15:09:01.959339 (+ 284us) server_negotiation.cc:299] Negotiation successful
0926 15:09:01.959560 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21650,"thread_start_us":21535,"threads_started":1}
W20260926 15:09:01.960515 14378 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59232
W20260926 15:09:01.961050 12427 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 14247: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 15:09:01.961308 12427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8uoJGv/build/asan/bin/kudu with pid 14247
I20260926 15:09:01.983541 12427 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 14387
Sep 26 15:09:01 dist-test-slave-865v krb5kdc[14131](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435339, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.12.34.194@KRBTEST.COM
I20260926 15:09:01.998543 14840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:01.984191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42476 (local address 127.12.34.194:41037)
0926 15:09:01.984739 (+ 548us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:01.984743 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:01.984757 (+ 14us) server_negotiation.cc:407] Connection header received
0926 15:09:01.984813 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:01.984817 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:01.984874 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:01.984986 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:09:01.986185 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:01.986927 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:01.988121 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:01.988523 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:01.988649 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:01.993000 (+ 4351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:09:01.993013 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:09:01.993030 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:09:01.993068 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:09:01.995855 (+ 2787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:01.996621 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:01.996627 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:01.996629 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:01.996741 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:01.997128 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:01.997132 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:01.997133 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:01.997483 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:09:01.997686 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0926 15:09:01.998096 (+ 410us) server_negotiation.cc:299] Negotiation successful
0926 15:09:01.998258 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":413,"thread_start_us":283,"threads_started":1}
W20260926 15:09:01.999235 14517 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42476
W20260926 15:09:01.999939 12427 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 14387: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 15:09:02.000136 12427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8uoJGv/build/asan/bin/kudu with pid 14387
I20260926 15:09:02.025633 12427 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 14526
Sep 26 15:09:02 dist-test-slave-865v krb5kdc[14131](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435339, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.12.34.195@KRBTEST.COM
I20260926 15:09:02.042152 14841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:02.026364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41610 (local address 127.12.34.195:39017)
0926 15:09:02.026895 (+ 531us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:02.026899 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:02.026916 (+ 17us) server_negotiation.cc:407] Connection header received
0926 15:09:02.026971 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:02.026974 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:02.027035 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:02.027130 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:09:02.028617 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:02.029501 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:02.030743 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:02.031191 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:02.031297 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:02.036082 (+ 4785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:09:02.036094 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:09:02.036110 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:09:02.036147 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:09:02.039341 (+ 3194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:02.040069 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:02.040084 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:02.040086 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:02.040216 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:02.040589 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:02.040597 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:02.040600 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:02.040946 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:09:02.041292 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0926 15:09:02.041568 (+ 276us) server_negotiation.cc:299] Negotiation successful
0926 15:09:02.041793 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":414,"thread_start_us":297,"threads_started":1}
W20260926 15:09:02.042815 14656 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41610
W20260926 15:09:02.043413 12427 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 14526: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 15:09:02.043598 12427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8uoJGv/build/asan/bin/kudu with pid 14526
I20260926 15:09:02.069214 12427 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 14144
I20260926 15:09:02.070505 14216 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:09:02.184934 12427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8uoJGv/build/asan/bin/kudu with pid 14144
2026-09-26T15:09:02Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8923 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435318278681-12427-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 15:09:02 dist-test-slave-865v krb5kdc[14845](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 15:09:02 dist-test-slave-865v krb5kdc[14845](info): set up 2 sockets
Sep 26 15:09:02 dist-test-slave-865v krb5kdc[14845](info): commencing operation
krb5kdc: starting...
W20260926 15:09:03.273411 12427 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.000s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435318278681-12427-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435318278681-12427-0/krb5kdc/test-user.keytab.
Sep 26 15:09:03 dist-test-slave-865v krb5kdc[14845](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435343, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T15:09:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T15:09:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.34.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.34.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.34.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.254.keytab.
Entry for principal kudu/127.12.34.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.254.keytab.
I20260926 15:09:03.464794 12427 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8uoJGv/build/asan/bin/kudu
/tmp/dist-test-task8uoJGv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435318278681-12427-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435318278681-12427-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435318278681-12427-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.34.254:46245
--webserver_interface=127.12.34.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.254.keytab
--principal=kudu/127.12.34.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.34.212:46277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.34.254:46245
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435318278681-12427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435318278681-12427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435318278681-12427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:09:03.789860 14858 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:09:03.790144 14858 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:09:03.790337 14858 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:09:03.797334 14858 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:09:03.797425 14858 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:09:03.797520 14858 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:09:03.797617 14858 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:09:03.797693 14858 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:09:03.807134 14858 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.34.212:46277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435318278681-12427-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435318278681-12427-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.34.254:46245
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.254.keytab
--principal=kudu/127.12.34.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.34.254:46245
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.34.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435318278681-12427-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:09:03.808400 14858 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:09:03.809664 14858 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:09:03.821286 14864 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:09:03.821580 14866 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:09:03.821609 14863 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:09:03.821949 14858 server_base.cc:1034] running on GCE node
I20260926 15:09:03.822712 14858 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:09:03.823995 14858 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:09:03.825172 14858 hybrid_clock.cc:648] HybridClock initialized: now 1790435343825132 us; error 61 us; skew 500 ppm
I20260926 15:09:03.825498 14858 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:09:03 dist-test-slave-865v krb5kdc[14845](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435343, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:09:03.828373 14858 init.cc:377] Logged in from keytab as kudu/127.12.34.254@KRBTEST.COM (short username kudu)
I20260926 15:09:03.829978 14858 webserver.cc:460] Webserver started at http://127.12.34.254:33453/ using document root <none> and password file <none>
I20260926 15:09:03.830621 14858 fs_manager.cc:359] Metadata directory not provided
I20260926 15:09:03.830753 14858 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:09:03.831053 14858 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:09:03.833869 14858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435318278681-12427-0/minicluster-data/master-0/data/instance:
uuid: "11f57715eff04fd59e99da0b438efb5a"
format_stamp: "Formatted at 2026-09-26 15:09:03 on dist-test-slave-865v"
server_key: "ac7b64136252f9f3c715f35326c29b9b"
server_key_iv: "2899f9ee201d5a4a25bf811c2e3c5b7d"
server_key_version: "encryptionkey@0"
I20260926 15:09:03.834605 14858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435318278681-12427-0/minicluster-data/master-0/wal/instance:
uuid: "11f57715eff04fd59e99da0b438efb5a"
format_stamp: "Formatted at 2026-09-26 15:09:03 on dist-test-slave-865v"
server_key: "ac7b64136252f9f3c715f35326c29b9b"
server_key_iv: "2899f9ee201d5a4a25bf811c2e3c5b7d"
server_key_version: "encryptionkey@0"
I20260926 15:09:03.839596 14858 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 15:09:03.842931 14873 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:09:03.844216 14858 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 15:09:03.844434 14858 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435318278681-12427-0/minicluster-data/master-0/data,/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435318278681-12427-0/minicluster-data/master-0/wal
uuid: "11f57715eff04fd59e99da0b438efb5a"
format_stamp: "Formatted at 2026-09-26 15:09:03 on dist-test-slave-865v"
server_key: "ac7b64136252f9f3c715f35326c29b9b"
server_key_iv: "2899f9ee201d5a4a25bf811c2e3c5b7d"
server_key_version: "encryptionkey@0"
I20260926 15:09:03.844637 14858 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435318278681-12427-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435318278681-12427-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435318278681-12427-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:09:03.864728 14858 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:09:03.868592 14858 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:09:03.868865 14858 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:09:03.886953 14858 rpc_server.cc:307] RPC server started. Bound to: 127.12.34.254:46245
I20260926 15:09:03.886986 14924 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.34.254:46245 every 8 connection(s)
I20260926 15:09:03.888262 14858 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
I20260926 15:09:03.891876 14925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:09:03.893574 12427 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8uoJGv/build/asan/bin/kudu as pid 14858
I20260926 15:09:03.893793 12427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435318278681-12427-0/minicluster-data/master-0/wal/instance
I20260926 15:09:03.894411 12427 external_mini_cluster.cc:1428] Setting key 86514e394878d3d9ed3fd9790ce8b1b1
I20260926 15:09:03.904412 14925 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 11f57715eff04fd59e99da0b438efb5a: Bootstrap starting.
I20260926 15:09:03.908017 14925 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 11f57715eff04fd59e99da0b438efb5a: Neither blocks nor log segments found. Creating new log.
I20260926 15:09:03.909008 14925 log.cc:824] T 00000000000000000000000000000000 P 11f57715eff04fd59e99da0b438efb5a: Log is configured to *not* fsync() on all Append() calls
Sep 26 15:09:03 dist-test-slave-865v krb5kdc[14845](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435343, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.34.254@KRBTEST.COM
I20260926 15:09:03.911860 14925 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 11f57715eff04fd59e99da0b438efb5a: No bootstrap required, opened a new log
I20260926 15:09:03.918143 14928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:03.896514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52838 (local address 127.12.34.254:46245)
0926 15:09:03.897329 (+ 815us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:03.897344 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:03.897400 (+ 56us) server_negotiation.cc:407] Connection header received
0926 15:09:03.899637 (+ 2237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:03.899662 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:03.900114 (+ 452us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:03.900630 (+ 516us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:09:03.902029 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:03.903944 (+ 1915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:03.905629 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:03.906695 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:03.906866 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:03.911086 (+ 4220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:09:03.911093 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:09:03.911108 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:09:03.911140 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:09:03.914225 (+ 3085us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:03.915028 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:03.915034 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:03.915039 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:03.915203 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:03.915577 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:03.915580 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:03.915582 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:03.916048 (+ 466us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:09:03.916285 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0926 15:09:03.916716 (+ 431us) server_negotiation.cc:299] Negotiation successful
0926 15:09:03.917139 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":562,"thread_start_us":333,"threads_started":1}
I20260926 15:09:03.920630 14925 raft_consensus.cc:348] T 00000000000000000000000000000000 P 11f57715eff04fd59e99da0b438efb5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11f57715eff04fd59e99da0b438efb5a" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 46245 } }
I20260926 15:09:03.921042 14925 raft_consensus.cc:374] T 00000000000000000000000000000000 P 11f57715eff04fd59e99da0b438efb5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:09:03.921164 14925 raft_consensus.cc:729] T 00000000000000000000000000000000 P 11f57715eff04fd59e99da0b438efb5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11f57715eff04fd59e99da0b438efb5a, State: Initialized, Role: FOLLOWER
I20260926 15:09:03.921649 14925 consensus_queue.cc:260] T 00000000000000000000000000000000 P 11f57715eff04fd59e99da0b438efb5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11f57715eff04fd59e99da0b438efb5a" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 46245 } }
I20260926 15:09:03.921835 14925 raft_consensus.cc:388] T 00000000000000000000000000000000 P 11f57715eff04fd59e99da0b438efb5a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 15:09:03.921923 14925 raft_consensus.cc:482] T 00000000000000000000000000000000 P 11f57715eff04fd59e99da0b438efb5a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 15:09:03.922042 14925 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 11f57715eff04fd59e99da0b438efb5a [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:09:03.923990 14925 raft_consensus.cc:504] T 00000000000000000000000000000000 P 11f57715eff04fd59e99da0b438efb5a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11f57715eff04fd59e99da0b438efb5a" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 46245 } }
I20260926 15:09:03.924389 14925 leader_election.cc:302] T 00000000000000000000000000000000 P 11f57715eff04fd59e99da0b438efb5a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 11f57715eff04fd59e99da0b438efb5a; no voters:
I20260926 15:09:03.924904 14925 leader_election.cc:288] T 00000000000000000000000000000000 P 11f57715eff04fd59e99da0b438efb5a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 15:09:03.925055 14930 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 11f57715eff04fd59e99da0b438efb5a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:09:03.925803 14930 raft_consensus.cc:686] T 00000000000000000000000000000000 P 11f57715eff04fd59e99da0b438efb5a [term 1 LEADER]: Becoming Leader. State: Replica: 11f57715eff04fd59e99da0b438efb5a, State: Running, Role: LEADER
I20260926 15:09:03.926334 14930 consensus_queue.cc:237] T 00000000000000000000000000000000 P 11f57715eff04fd59e99da0b438efb5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11f57715eff04fd59e99da0b438efb5a" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 46245 } }
I20260926 15:09:03.927011 14925 sys_catalog.cc:564] T 00000000000000000000000000000000 P 11f57715eff04fd59e99da0b438efb5a [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:09:03.929111 14931 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11f57715eff04fd59e99da0b438efb5a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "11f57715eff04fd59e99da0b438efb5a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11f57715eff04fd59e99da0b438efb5a" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 46245 } } }
I20260926 15:09:03.929165 14932 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11f57715eff04fd59e99da0b438efb5a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 11f57715eff04fd59e99da0b438efb5a. Latest consensus state: current_term: 1 leader_uuid: "11f57715eff04fd59e99da0b438efb5a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11f57715eff04fd59e99da0b438efb5a" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 46245 } } }
I20260926 15:09:03.929563 14932 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11f57715eff04fd59e99da0b438efb5a [sys.catalog]: This master's current role is: LEADER
I20260926 15:09:03.929557 14931 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11f57715eff04fd59e99da0b438efb5a [sys.catalog]: This master's current role is: LEADER
I20260926 15:09:03.931384 14936 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:09:03.936275 14936 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 15:09:03.945237 14936 catalog_manager.cc:1347] Generated new cluster ID: 00b0ab03de0b4ad3b2d3e307248ea6cc
I20260926 15:09:03.945351 14936 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 15:09:03.962942 14936 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 15:09:03.964177 14936 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:09:03.973623 14936 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 11f57715eff04fd59e99da0b438efb5a: Generated new TSK 0
I20260926 15:09:03.974402 14936 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.34.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.34.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.34.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.193.keytab.
Entry for principal kudu/127.12.34.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.193.keytab.
I20260926 15:09:04.025326 12427 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8uoJGv/build/asan/bin/kudu
/tmp/dist-test-task8uoJGv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435318278681-12427-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435318278681-12427-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435318278681-12427-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435318278681-12427-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.34.193:0
--local_ip_for_outbound_sockets=127.12.34.193
--webserver_interface=127.12.34.193
--webserver_port=0
--tserver_master_addrs=127.12.34.254:46245
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.193.keytab
--principal=kudu/127.12.34.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.34.212:46277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435318278681-12427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435318278681-12427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435318278681-12427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:09:04.374536 14951 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:09:04.374786 14951 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:09:04.374953 14951 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:09:04.381896 14951 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:09:04.381992 14951 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:09:04.382174 14951 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.34.193
I20260926 15:09:04.391348 14951 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.34.212:46277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435318278681-12427-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435318278681-12427-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.193.keytab
--principal=kudu/127.12.34.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.34.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435318278681-12427-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.34.193
--webserver_port=0
--tserver_master_addrs=127.12.34.254:46245
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.34.193
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435318278681-12427-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:09:04.392508 14951 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:09:04.393755 14951 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:09:04.406069 14957 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:09:04.406108 14959 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:09:04.406572 14956 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:09:04.407182 14951 server_base.cc:1034] running on GCE node
I20260926 15:09:04.407809 14951 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:09:04.408726 14951 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:09:04.409929 14951 hybrid_clock.cc:648] HybridClock initialized: now 1790435344409865 us; error 50 us; skew 500 ppm
I20260926 15:09:04.410264 14951 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:09:04 dist-test-slave-865v krb5kdc[14845](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435344, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:09:04.413386 14951 init.cc:377] Logged in from keytab as kudu/127.12.34.193@KRBTEST.COM (short username kudu)
I20260926 15:09:04.414757 14951 webserver.cc:460] Webserver started at http://127.12.34.193:42225/ using document root <none> and password file <none>
I20260926 15:09:04.415359 14951 fs_manager.cc:359] Metadata directory not provided
I20260926 15:09:04.415452 14951 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:09:04.415707 14951 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:09:04.418618 14951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435318278681-12427-0/minicluster-data/ts-0/data/instance:
uuid: "fa24c64ced874aebb551388121361f6d"
format_stamp: "Formatted at 2026-09-26 15:09:04 on dist-test-slave-865v"
server_key: "1e7a64111b9a03a1b754a39e9451286e"
server_key_iv: "bb8c7abc77eb963ccdde24dbd15a02ea"
server_key_version: "encryptionkey@0"
I20260926 15:09:04.419304 14951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435318278681-12427-0/minicluster-data/ts-0/wal/instance:
uuid: "fa24c64ced874aebb551388121361f6d"
format_stamp: "Formatted at 2026-09-26 15:09:04 on dist-test-slave-865v"
server_key: "1e7a64111b9a03a1b754a39e9451286e"
server_key_iv: "bb8c7abc77eb963ccdde24dbd15a02ea"
server_key_version: "encryptionkey@0"
I20260926 15:09:04.424108 14951 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260926 15:09:04.427358 14966 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:09:04.428737 14951 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260926 15:09:04.428860 14951 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435318278681-12427-0/minicluster-data/ts-0/data,/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435318278681-12427-0/minicluster-data/ts-0/wal
uuid: "fa24c64ced874aebb551388121361f6d"
format_stamp: "Formatted at 2026-09-26 15:09:04 on dist-test-slave-865v"
server_key: "1e7a64111b9a03a1b754a39e9451286e"
server_key_iv: "bb8c7abc77eb963ccdde24dbd15a02ea"
server_key_version: "encryptionkey@0"
I20260926 15:09:04.429057 14951 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435318278681-12427-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435318278681-12427-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435318278681-12427-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:09:04.454862 14951 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:09:04.458882 14951 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:09:04.459117 14951 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:09:04.459975 14951 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:09:04.461311 14951 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:09:04.461382 14951 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:09:04.461464 14951 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:09:04.461515 14951 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:09:04.494333 14951 rpc_server.cc:307] RPC server started. Bound to: 127.12.34.193:42501
I20260926 15:09:04.494415 15078 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.34.193:42501 every 8 connection(s)
I20260926 15:09:04.495581 14951 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435318278681-12427-0/minicluster-data/ts-0/data/info.pb
I20260926 15:09:04.505795 12427 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8uoJGv/build/asan/bin/kudu as pid 14951
I20260926 15:09:04.506026 12427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435318278681-12427-0/minicluster-data/ts-0/wal/instance
I20260926 15:09:04.506784 12427 external_mini_cluster.cc:1428] Setting key 34504e3b31b0298b9d7e89b4be7b0244
Sep 26 15:09:04 dist-test-slave-865v krb5kdc[14845](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435344, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.193@KRBTEST.COM for kudu/127.12.34.254@KRBTEST.COM
I20260926 15:09:04.517642 15081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:04.498208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.193:34563 (local address 127.12.34.254:46245)
0926 15:09:04.498630 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:04.498635 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:04.499611 (+ 976us) server_negotiation.cc:407] Connection header received
0926 15:09:04.501449 (+ 1838us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:04.501455 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:04.501551 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:04.501669 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:09:04.503845 (+ 2176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:04.504563 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:04.506741 (+ 2178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:04.507251 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:04.507429 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:04.511476 (+ 4047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:09:04.511481 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:09:04.511485 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:09:04.511515 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:09:04.514246 (+ 2731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:04.515279 (+ 1033us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:04.515283 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:04.515285 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:04.515411 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:04.515903 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:04.515907 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:04.515909 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:04.516166 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:09:04.516281 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0926 15:09:04.517206 (+ 925us) server_negotiation.cc:299] Negotiation successful
0926 15:09:04.517375 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":212,"threads_started":1}
I20260926 15:09:04.519071 15082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:04.498756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:46245 (local address 127.12.34.193:34563)
0926 15:09:04.499434 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0926 15:09:04.499474 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 15:09:04.501194 (+ 1720us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:09:04.501935 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:09:04.501947 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:09:04.502774 (+ 827us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:09:04.503678 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:04.503705 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:04.504707 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:04.504712 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:04.506551 (+ 1839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:04.506568 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:04.507535 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:04.507543 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:04.507821 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:04.508591 (+ 770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:09:04.508621 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:09:04.511262 (+ 2641us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:09:04.514387 (+ 3125us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:09:04.514398 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:09:04.514418 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:09:04.515052 (+ 634us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:09:04.515519 (+ 467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:09:04.515526 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:09:04.515533 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:09:04.515760 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:09:04.516296 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:09:04.516306 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:09:04.516743 (+ 437us) client_negotiation.cc:769] Sending connection context
0926 15:09:04.517058 (+ 315us) client_negotiation.cc:240] Negotiation successful
0926 15:09:04.517636 (+ 578us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":281,"threads_started":1}
I20260926 15:09:04.520571 15079 heartbeater.cc:347] Connected to a master server at 127.12.34.254:46245
I20260926 15:09:04.520864 15079 heartbeater.cc:464] Registering TS with master...
I20260926 15:09:04.521675 15079 heartbeater.cc:511] Master 127.12.34.254:46245 requested a full tablet report, sending...
I20260926 15:09:04.523728 14890 ts_manager.cc:194] Registered new tserver with Master: fa24c64ced874aebb551388121361f6d (127.12.34.193:42501)
WARNING: no policy specified for kudu/127.12.34.194@KRBTEST.COM; defaulting to no policy
I20260926 15:09:04.525254 14890 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.34.193@KRBTEST.COM'} at 127.12.34.193:34563
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.34.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.34.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.194.keytab.
Entry for principal kudu/127.12.34.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.194.keytab.
I20260926 15:09:04.552163 12427 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8uoJGv/build/asan/bin/kudu
/tmp/dist-test-task8uoJGv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435318278681-12427-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435318278681-12427-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435318278681-12427-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435318278681-12427-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.34.194:0
--local_ip_for_outbound_sockets=127.12.34.194
--webserver_interface=127.12.34.194
--webserver_port=0
--tserver_master_addrs=127.12.34.254:46245
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.194.keytab
--principal=kudu/127.12.34.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.34.212:46277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435318278681-12427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435318278681-12427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435318278681-12427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:09:04.868569 15085 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:09:04.868860 15085 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:09:04.869035 15085 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:09:04.875980 15085 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:09:04.876120 15085 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:09:04.876302 15085 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.34.194
I20260926 15:09:04.885705 15085 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.34.212:46277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435318278681-12427-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435318278681-12427-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.194.keytab
--principal=kudu/127.12.34.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.34.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435318278681-12427-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.34.194
--webserver_port=0
--tserver_master_addrs=127.12.34.254:46245
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.34.194
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435318278681-12427-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:09:04.887265 15085 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:09:04.888847 15085 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:09:04.903007 15090 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:09:04.903117 15091 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:09:04.903648 15085 server_base.cc:1034] running on GCE node
W20260926 15:09:04.903734 15093 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:09:04.904363 15085 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:09:04.905261 15085 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:09:04.906508 15085 hybrid_clock.cc:648] HybridClock initialized: now 1790435344906456 us; error 40 us; skew 500 ppm
I20260926 15:09:04.906843 15085 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:09:04 dist-test-slave-865v krb5kdc[14845](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435344, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:09:04.909890 15085 init.cc:377] Logged in from keytab as kudu/127.12.34.194@KRBTEST.COM (short username kudu)
I20260926 15:09:04.911240 15085 webserver.cc:460] Webserver started at http://127.12.34.194:46195/ using document root <none> and password file <none>
I20260926 15:09:04.911914 15085 fs_manager.cc:359] Metadata directory not provided
I20260926 15:09:04.912057 15085 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:09:04.912343 15085 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:09:04.915069 15085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435318278681-12427-0/minicluster-data/ts-1/data/instance:
uuid: "7fbb2fce53534302ab8be9875fd00543"
format_stamp: "Formatted at 2026-09-26 15:09:04 on dist-test-slave-865v"
server_key: "1a58ea1a68afc69708350534f9ed5644"
server_key_iv: "7ec63d534a0d2cfe04f1c51c3b92fa37"
server_key_version: "encryptionkey@0"
I20260926 15:09:04.915803 15085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435318278681-12427-0/minicluster-data/ts-1/wal/instance:
uuid: "7fbb2fce53534302ab8be9875fd00543"
format_stamp: "Formatted at 2026-09-26 15:09:04 on dist-test-slave-865v"
server_key: "1a58ea1a68afc69708350534f9ed5644"
server_key_iv: "7ec63d534a0d2cfe04f1c51c3b92fa37"
server_key_version: "encryptionkey@0"
I20260926 15:09:04.920863 15085 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260926 15:09:04.924423 15100 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:09:04.925958 15085 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 15:09:04.926128 15085 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435318278681-12427-0/minicluster-data/ts-1/data,/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435318278681-12427-0/minicluster-data/ts-1/wal
uuid: "7fbb2fce53534302ab8be9875fd00543"
format_stamp: "Formatted at 2026-09-26 15:09:04 on dist-test-slave-865v"
server_key: "1a58ea1a68afc69708350534f9ed5644"
server_key_iv: "7ec63d534a0d2cfe04f1c51c3b92fa37"
server_key_version: "encryptionkey@0"
I20260926 15:09:04.926324 15085 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435318278681-12427-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435318278681-12427-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435318278681-12427-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:09:04.944590 15085 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:09:04.949373 15085 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:09:04.949657 15085 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:09:04.950609 15085 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:09:04.952107 15085 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:09:04.952220 15085 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:09:04.952389 15085 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:09:04.952564 15085 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:09:04.989728 15085 rpc_server.cc:307] RPC server started. Bound to: 127.12.34.194:42365
I20260926 15:09:04.989825 15212 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.34.194:42365 every 8 connection(s)
I20260926 15:09:04.990963 15085 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435318278681-12427-0/minicluster-data/ts-1/data/info.pb
I20260926 15:09:04.999639 12427 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8uoJGv/build/asan/bin/kudu as pid 15085
I20260926 15:09:04.999836 12427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435318278681-12427-0/minicluster-data/ts-1/wal/instance
I20260926 15:09:05.000247 12427 external_mini_cluster.cc:1428] Setting key 3072c0304285ecbd221f2f1ed3c77c6e
Sep 26 15:09:05 dist-test-slave-865v krb5kdc[14845](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435344, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.194@KRBTEST.COM for kudu/127.12.34.254@KRBTEST.COM
I20260926 15:09:05.012873 15081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:04.994334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.194:55655 (local address 127.12.34.254:46245)
0926 15:09:04.994570 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:04.994575 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:04.995754 (+ 1179us) server_negotiation.cc:407] Connection header received
0926 15:09:04.997569 (+ 1815us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:04.997574 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:04.997668 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:04.997780 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:09:04.999190 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:04.999982 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:05.001810 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:05.002426 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:05.002566 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:05.006675 (+ 4109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:09:05.006684 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:09:05.006692 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:09:05.006738 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:09:05.009511 (+ 2773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:05.010481 (+ 970us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:05.010485 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:05.010488 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:05.010628 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:05.011144 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:05.011148 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:05.011151 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:05.011408 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:09:05.011525 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0926 15:09:05.012429 (+ 904us) server_negotiation.cc:299] Negotiation successful
0926 15:09:05.012627 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260926 15:09:05.014365 15215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:04.994767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:46245 (local address 127.12.34.194:55655)
0926 15:09:04.995531 (+ 764us) negotiation.cc:107] Waiting for socket to connect
0926 15:09:04.995575 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 15:09:04.997263 (+ 1688us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:09:04.998009 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:09:04.998018 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:09:04.998491 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:09:04.999028 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:04.999044 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:05.000130 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:05.000134 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:05.001644 (+ 1510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:05.001656 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:05.002625 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:05.002630 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:05.002806 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:05.003873 (+ 1067us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:09:05.003897 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:09:05.006442 (+ 2545us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:09:05.009641 (+ 3199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:09:05.009652 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:09:05.009671 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:09:05.010321 (+ 650us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:09:05.010737 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:09:05.010744 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:09:05.010751 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:09:05.010995 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:09:05.011539 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:09:05.011549 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:09:05.012005 (+ 456us) client_negotiation.cc:769] Sending connection context
0926 15:09:05.012306 (+ 301us) client_negotiation.cc:240] Negotiation successful
0926 15:09:05.012871 (+ 565us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":296,"threads_started":1}
I20260926 15:09:05.015969 15213 heartbeater.cc:347] Connected to a master server at 127.12.34.254:46245
I20260926 15:09:05.016317 15213 heartbeater.cc:464] Registering TS with master...
I20260926 15:09:05.017234 15213 heartbeater.cc:511] Master 127.12.34.254:46245 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.12.34.195@KRBTEST.COM; defaulting to no policy
I20260926 15:09:05.019132 14890 ts_manager.cc:194] Registered new tserver with Master: 7fbb2fce53534302ab8be9875fd00543 (127.12.34.194:42365)
I20260926 15:09:05.020356 14890 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.34.194@KRBTEST.COM'} at 127.12.34.194:55655
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.34.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.34.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.195.keytab.
Entry for principal kudu/127.12.34.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.195.keytab.
I20260926 15:09:05.050063 12427 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8uoJGv/build/asan/bin/kudu
/tmp/dist-test-task8uoJGv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435318278681-12427-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435318278681-12427-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435318278681-12427-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435318278681-12427-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.34.195:0
--local_ip_for_outbound_sockets=127.12.34.195
--webserver_interface=127.12.34.195
--webserver_port=0
--tserver_master_addrs=127.12.34.254:46245
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.195.keytab
--principal=kudu/127.12.34.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.34.212:46277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435318278681-12427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435318278681-12427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435318278681-12427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:09:05.390480 15218 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:09:05.390795 15218 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:09:05.390960 15218 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:09:05.398108 15218 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:09:05.398262 15218 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:09:05.398447 15218 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.34.195
I20260926 15:09:05.408246 15218 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.34.212:46277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435318278681-12427-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435318278681-12427-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.195.keytab
--principal=kudu/127.12.34.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.34.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435318278681-12427-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.34.195
--webserver_port=0
--tserver_master_addrs=127.12.34.254:46245
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.34.195
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435318278681-12427-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:09:05.409626 15218 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:09:05.410900 15218 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:09:05.423332 15223 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:09:05.423403 15224 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:09:05.423349 15226 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:09:05.424521 15218 server_base.cc:1034] running on GCE node
I20260926 15:09:05.425072 15218 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:09:05.425985 15218 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:09:05.427208 15218 hybrid_clock.cc:648] HybridClock initialized: now 1790435345427176 us; error 43 us; skew 500 ppm
I20260926 15:09:05.427551 15218 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:09:05 dist-test-slave-865v krb5kdc[14845](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435345, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:09:05.430755 15218 init.cc:377] Logged in from keytab as kudu/127.12.34.195@KRBTEST.COM (short username kudu)
I20260926 15:09:05.432339 15218 webserver.cc:460] Webserver started at http://127.12.34.195:34055/ using document root <none> and password file <none>
I20260926 15:09:05.433040 15218 fs_manager.cc:359] Metadata directory not provided
I20260926 15:09:05.433230 15218 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:09:05.433538 15218 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:09:05.436318 15218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435318278681-12427-0/minicluster-data/ts-2/data/instance:
uuid: "183e342c584f45a3a9a35bfc27644b9f"
format_stamp: "Formatted at 2026-09-26 15:09:05 on dist-test-slave-865v"
server_key: "69ff5c4d2913f8eed410e8a1f25c8257"
server_key_iv: "07a7c9e7ea7e47843a6ea3cef3fae636"
server_key_version: "encryptionkey@0"
I20260926 15:09:05.437070 15218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435318278681-12427-0/minicluster-data/ts-2/wal/instance:
uuid: "183e342c584f45a3a9a35bfc27644b9f"
format_stamp: "Formatted at 2026-09-26 15:09:05 on dist-test-slave-865v"
server_key: "69ff5c4d2913f8eed410e8a1f25c8257"
server_key_iv: "07a7c9e7ea7e47843a6ea3cef3fae636"
server_key_version: "encryptionkey@0"
I20260926 15:09:05.442312 15218 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260926 15:09:05.445600 15233 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:09:05.446832 15218 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 15:09:05.447013 15218 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435318278681-12427-0/minicluster-data/ts-2/data,/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435318278681-12427-0/minicluster-data/ts-2/wal
uuid: "183e342c584f45a3a9a35bfc27644b9f"
format_stamp: "Formatted at 2026-09-26 15:09:05 on dist-test-slave-865v"
server_key: "69ff5c4d2913f8eed410e8a1f25c8257"
server_key_iv: "07a7c9e7ea7e47843a6ea3cef3fae636"
server_key_version: "encryptionkey@0"
I20260926 15:09:05.447204 15218 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435318278681-12427-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435318278681-12427-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435318278681-12427-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:09:05.465219 15218 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:09:05.468971 15218 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:09:05.469267 15218 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:09:05.470142 15218 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:09:05.471424 15218 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:09:05.471493 15218 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:09:05.471617 15218 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:09:05.471668 15218 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:09:05.506604 15218 rpc_server.cc:307] RPC server started. Bound to: 127.12.34.195:42449
I20260926 15:09:05.506636 15345 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.34.195:42449 every 8 connection(s)
I20260926 15:09:05.507822 15218 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435318278681-12427-0/minicluster-data/ts-2/data/info.pb
I20260926 15:09:05.510627 12427 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8uoJGv/build/asan/bin/kudu as pid 15218
I20260926 15:09:05.510823 12427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790435318278681-12427-0/minicluster-data/ts-2/wal/instance
I20260926 15:09:05.511238 12427 external_mini_cluster.cc:1428] Setting key 43d576670339d2c4fe3ac28bd876a87d
Sep 26 15:09:05 dist-test-slave-865v krb5kdc[14845](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435345, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.195@KRBTEST.COM for kudu/127.12.34.254@KRBTEST.COM
I20260926 15:09:05.527937 15081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:05.510414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.195:55253 (local address 127.12.34.254:46245)
0926 15:09:05.510631 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:05.510637 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:05.511933 (+ 1296us) server_negotiation.cc:407] Connection header received
0926 15:09:05.513822 (+ 1889us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:05.513827 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:05.513909 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:05.514022 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:09:05.515423 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:05.516156 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:05.517704 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:05.518099 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:05.518232 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:05.521839 (+ 3607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:09:05.521845 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:09:05.521849 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:09:05.521882 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:09:05.524747 (+ 2865us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:05.525696 (+ 949us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:05.525700 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:05.525703 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:05.525901 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:05.526416 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:05.526420 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:05.526422 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:05.526737 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:09:05.526845 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 15:09:05.527480 (+ 635us) server_negotiation.cc:299] Negotiation successful
0926 15:09:05.527650 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260926 15:09:05.528371 15079 heartbeater.cc:503] Master 127.12.34.254:46245 was elected leader, sending a full tablet report...
I20260926 15:09:05.529203 15348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:05.510971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:46245 (local address 127.12.34.195:55253)
0926 15:09:05.511776 (+ 805us) negotiation.cc:107] Waiting for socket to connect
0926 15:09:05.511828 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 15:09:05.513573 (+ 1745us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:09:05.514241 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:09:05.514248 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:09:05.514724 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:09:05.515255 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:05.515270 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:05.516301 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:05.516307 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:05.517560 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:05.517572 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:05.518319 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:05.518327 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:05.518557 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:05.519211 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:09:05.519238 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:09:05.521620 (+ 2382us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:09:05.524918 (+ 3298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:09:05.524927 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:09:05.524942 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:09:05.525531 (+ 589us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:09:05.526027 (+ 496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:09:05.526032 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:09:05.526035 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:09:05.526288 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:09:05.526852 (+ 564us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:09:05.526858 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:09:05.527156 (+ 298us) client_negotiation.cc:769] Sending connection context
0926 15:09:05.527393 (+ 237us) client_negotiation.cc:240] Negotiation successful
0926 15:09:05.527835 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":559,"thread_start_us":342,"threads_started":1}
I20260926 15:09:05.530651 15346 heartbeater.cc:347] Connected to a master server at 127.12.34.254:46245
I20260926 15:09:05.530969 15346 heartbeater.cc:464] Registering TS with master...
I20260926 15:09:05.531651 15346 heartbeater.cc:511] Master 127.12.34.254:46245 requested a full tablet report, sending...
I20260926 15:09:05.533135 14890 ts_manager.cc:194] Registered new tserver with Master: 183e342c584f45a3a9a35bfc27644b9f (127.12.34.195:42449)
I20260926 15:09:05.533905 14890 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.34.195@KRBTEST.COM'} at 127.12.34.195:55253
I20260926 15:09:05.538709 12427 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 15:09:05.569310 15081 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:09:05.566877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52846 (local address 127.12.34.254:46245)
0926 15:09:05.567083 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:05.567088 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:05.567593 (+ 505us) server_negotiation.cc:407] Connection header received
0926 15:09:05.567726 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:05.567731 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:05.567799 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:05.567897 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:09:05.568982 (+ 1085us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52846: BlockingRecv error: recv got EOF from 127.0.0.1:52846 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
I20260926 15:09:05.572525 12427 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 14951
W20260926 15:09:05.575367 12427 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 14951: Not authorized: Client connection negotiation failed: client connection to 127.12.34.193:42501: server requires authentication, but client does not have Kerberos credentials available
I20260926 15:09:05.575542 12427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8uoJGv/build/asan/bin/kudu with pid 14951
I20260926 15:09:05.593729 12427 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 15085
W20260926 15:09:05.596174 12427 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 15085: Not authorized: Client connection negotiation failed: client connection to 127.12.34.194:42365: server requires authentication, but client does not have Kerberos credentials available
I20260926 15:09:05.596335 12427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8uoJGv/build/asan/bin/kudu with pid 15085
I20260926 15:09:05.611284 12427 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 15218
W20260926 15:09:05.614105 12427 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 15218: Not authorized: Client connection negotiation failed: client connection to 127.12.34.195:42449: server requires authentication, but client does not have Kerberos credentials available
I20260926 15:09:05.614224 12427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8uoJGv/build/asan/bin/kudu with pid 15218
I20260926 15:09:05.630425 12427 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 14858
I20260926 15:09:05.631603 14920 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:09:05.736167 12427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8uoJGv/build/asan/bin/kudu with pid 14858
2026-09-26T15:09:05Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3547 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790435318278681-12427-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 15:09:05 dist-test-slave-865v krb5kdc[15363](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 15:09:05 dist-test-slave-865v krb5kdc[15363](info): set up 2 sockets
Sep 26 15:09:05 dist-test-slave-865v krb5kdc[15363](info): commencing operation
krb5kdc: starting...
W20260926 15:09:06.841643 12427 mini_kdc.cc:121] Time spent starting KDC: real 1.064s user 0.004s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790435318278681-12427-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790435318278681-12427-0/krb5kdc/test-user.keytab.
Sep 26 15:09:06 dist-test-slave-865v krb5kdc[15363](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435346, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T15:09:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T15:09:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.34.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.34.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.34.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.254.keytab.
Entry for principal kudu/127.12.34.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.254.keytab.
I20260926 15:09:07.054630 12427 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8uoJGv/build/asan/bin/kudu
/tmp/dist-test-task8uoJGv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790435318278681-12427-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790435318278681-12427-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790435318278681-12427-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.34.254:34111
--webserver_interface=127.12.34.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.254.keytab
--principal=kudu/127.12.34.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.34.212:39193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.34.254:34111
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790435318278681-12427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790435318278681-12427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790435318278681-12427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:09:07.379993 15376 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:09:07.380321 15376 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:09:07.380511 15376 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:09:07.387811 15376 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:09:07.387954 15376 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:09:07.388062 15376 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:09:07.388165 15376 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:09:07.388254 15376 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:09:07.397850 15376 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.34.212:39193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790435318278681-12427-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790435318278681-12427-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.34.254:34111
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.254.keytab
--principal=kudu/127.12.34.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.34.254:34111
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.34.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790435318278681-12427-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:09:07.399083 15376 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:09:07.400296 15376 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:09:07.412050 15381 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:09:07.411988 15384 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:09:07.411983 15382 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:09:07.413376 15376 server_base.cc:1034] running on GCE node
I20260926 15:09:07.414110 15376 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:09:07.415702 15376 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:09:07.417016 15376 hybrid_clock.cc:648] HybridClock initialized: now 1790435347416964 us; error 59 us; skew 500 ppm
I20260926 15:09:07.417425 15376 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:09:07 dist-test-slave-865v krb5kdc[15363](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435347, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:09:07.427002 15376 init.cc:377] Logged in from keytab as kudu/127.12.34.254@KRBTEST.COM (short username kudu)
I20260926 15:09:07.428601 15376 webserver.cc:460] Webserver started at http://127.12.34.254:41363/ using document root <none> and password file <none>
I20260926 15:09:07.429342 15376 fs_manager.cc:359] Metadata directory not provided
I20260926 15:09:07.429447 15376 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:09:07.429698 15376 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:09:07.432610 15376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790435318278681-12427-0/minicluster-data/master-0/data/instance:
uuid: "f5fe0e3a202c4384b01c4ef139842702"
format_stamp: "Formatted at 2026-09-26 15:09:07 on dist-test-slave-865v"
server_key: "8888380fd1843b4a9ae24caac82860e4"
server_key_iv: "921d30da5c3c41d15dde09db15c3f4ce"
server_key_version: "encryptionkey@0"
I20260926 15:09:07.433419 15376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790435318278681-12427-0/minicluster-data/master-0/wal/instance:
uuid: "f5fe0e3a202c4384b01c4ef139842702"
format_stamp: "Formatted at 2026-09-26 15:09:07 on dist-test-slave-865v"
server_key: "8888380fd1843b4a9ae24caac82860e4"
server_key_iv: "921d30da5c3c41d15dde09db15c3f4ce"
server_key_version: "encryptionkey@0"
I20260926 15:09:07.438699 15376 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260926 15:09:07.442129 15391 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:09:07.443418 15376 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260926 15:09:07.443607 15376 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790435318278681-12427-0/minicluster-data/master-0/data,/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790435318278681-12427-0/minicluster-data/master-0/wal
uuid: "f5fe0e3a202c4384b01c4ef139842702"
format_stamp: "Formatted at 2026-09-26 15:09:07 on dist-test-slave-865v"
server_key: "8888380fd1843b4a9ae24caac82860e4"
server_key_iv: "921d30da5c3c41d15dde09db15c3f4ce"
server_key_version: "encryptionkey@0"
I20260926 15:09:07.443768 15376 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790435318278681-12427-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790435318278681-12427-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790435318278681-12427-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:09:07.477671 15376 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:09:07.482752 15376 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:09:07.483002 15376 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:09:07.503772 15376 rpc_server.cc:307] RPC server started. Bound to: 127.12.34.254:34111
I20260926 15:09:07.503875 15442 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.34.254:34111 every 8 connection(s)
I20260926 15:09:07.505510 15376 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
I20260926 15:09:07.506724 12427 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8uoJGv/build/asan/bin/kudu as pid 15376
I20260926 15:09:07.506973 12427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790435318278681-12427-0/minicluster-data/master-0/wal/instance
I20260926 15:09:07.507640 12427 external_mini_cluster.cc:1428] Setting key a2a21225fbae1160b0c86680e2024ace
I20260926 15:09:07.510555 15443 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 26 15:09:07 dist-test-slave-865v krb5kdc[15363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435346, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.34.254@KRBTEST.COM
I20260926 15:09:07.523833 15443 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5fe0e3a202c4384b01c4ef139842702: Bootstrap starting.
I20260926 15:09:07.527207 15443 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f5fe0e3a202c4384b01c4ef139842702: Neither blocks nor log segments found. Creating new log.
I20260926 15:09:07.528451 15443 log.cc:824] T 00000000000000000000000000000000 P f5fe0e3a202c4384b01c4ef139842702: Log is configured to *not* fsync() on all Append() calls
I20260926 15:09:07.531642 15443 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5fe0e3a202c4384b01c4ef139842702: No bootstrap required, opened a new log
I20260926 15:09:07.532178 15446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:07.509995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37246 (local address 127.12.34.254:34111)
0926 15:09:07.510882 (+ 887us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:07.510895 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:07.510939 (+ 44us) server_negotiation.cc:407] Connection header received
0926 15:09:07.512692 (+ 1753us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:07.512714 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:07.513214 (+ 500us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:07.513578 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:09:07.515432 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:07.516811 (+ 1379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:07.518559 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:07.519220 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:07.519364 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:07.524116 (+ 4752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:09:07.524125 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:09:07.524139 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:09:07.524179 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:09:07.527353 (+ 3174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:07.528223 (+ 870us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:07.528238 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:07.528244 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:07.528412 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:07.528852 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:07.528857 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:07.528859 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:07.529717 (+ 858us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:09:07.529996 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0926 15:09:07.530588 (+ 592us) server_negotiation.cc:299] Negotiation successful
0926 15:09:07.531059 (+ 471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":657,"thread_start_us":429,"threads_started":1}
I20260926 15:09:07.536649 12427 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 15:09:07.541121 15443 raft_consensus.cc:348] T 00000000000000000000000000000000 P f5fe0e3a202c4384b01c4ef139842702 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5fe0e3a202c4384b01c4ef139842702" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 34111 } }
I20260926 15:09:07.541590 15443 raft_consensus.cc:374] T 00000000000000000000000000000000 P f5fe0e3a202c4384b01c4ef139842702 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:09:07.541697 15443 raft_consensus.cc:729] T 00000000000000000000000000000000 P f5fe0e3a202c4384b01c4ef139842702 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5fe0e3a202c4384b01c4ef139842702, State: Initialized, Role: FOLLOWER
I20260926 15:09:07.542356 15443 consensus_queue.cc:260] T 00000000000000000000000000000000 P f5fe0e3a202c4384b01c4ef139842702 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5fe0e3a202c4384b01c4ef139842702" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 34111 } }
I20260926 15:09:07.542667 15443 raft_consensus.cc:388] T 00000000000000000000000000000000 P f5fe0e3a202c4384b01c4ef139842702 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 15:09:07.542812 15443 raft_consensus.cc:482] T 00000000000000000000000000000000 P f5fe0e3a202c4384b01c4ef139842702 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 15:09:07.542984 15443 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f5fe0e3a202c4384b01c4ef139842702 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:09:07.545677 15443 raft_consensus.cc:504] T 00000000000000000000000000000000 P f5fe0e3a202c4384b01c4ef139842702 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5fe0e3a202c4384b01c4ef139842702" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 34111 } }
I20260926 15:09:07.546172 15443 leader_election.cc:302] T 00000000000000000000000000000000 P f5fe0e3a202c4384b01c4ef139842702 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f5fe0e3a202c4384b01c4ef139842702; no voters:
I20260926 15:09:07.546826 15443 leader_election.cc:288] T 00000000000000000000000000000000 P f5fe0e3a202c4384b01c4ef139842702 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 15:09:07.546945 15449 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f5fe0e3a202c4384b01c4ef139842702 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:09:07.549574 15443 sys_catalog.cc:564] T 00000000000000000000000000000000 P f5fe0e3a202c4384b01c4ef139842702 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:09:07.553409 15449 raft_consensus.cc:686] T 00000000000000000000000000000000 P f5fe0e3a202c4384b01c4ef139842702 [term 1 LEADER]: Becoming Leader. State: Replica: f5fe0e3a202c4384b01c4ef139842702, State: Running, Role: LEADER
I20260926 15:09:07.554214 15449 consensus_queue.cc:237] T 00000000000000000000000000000000 P f5fe0e3a202c4384b01c4ef139842702 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5fe0e3a202c4384b01c4ef139842702" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 34111 } }
I20260926 15:09:07.557459 15450 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5fe0e3a202c4384b01c4ef139842702 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f5fe0e3a202c4384b01c4ef139842702" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5fe0e3a202c4384b01c4ef139842702" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 34111 } } }
I20260926 15:09:07.557924 15450 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5fe0e3a202c4384b01c4ef139842702 [sys.catalog]: This master's current role is: LEADER
I20260926 15:09:07.558583 15451 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5fe0e3a202c4384b01c4ef139842702 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f5fe0e3a202c4384b01c4ef139842702. Latest consensus state: current_term: 1 leader_uuid: "f5fe0e3a202c4384b01c4ef139842702" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5fe0e3a202c4384b01c4ef139842702" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 34111 } } }
I20260926 15:09:07.559307 15451 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5fe0e3a202c4384b01c4ef139842702 [sys.catalog]: This master's current role is: LEADER
I20260926 15:09:07.572930 15465 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:09:07.578290 15465 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 15:09:07.587759 15465 catalog_manager.cc:1347] Generated new cluster ID: 68c7c574c6f0491a8432f374e1f1ee97
I20260926 15:09:07.587872 15465 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 15:09:07.590873 15446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:07.583444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37254 (local address 127.12.34.254:34111)
0926 15:09:07.583602 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:07.583607 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:07.583624 (+ 17us) server_negotiation.cc:407] Connection header received
0926 15:09:07.583671 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:07.583674 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:07.583731 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:07.583819 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:09:07.584959 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:07.586002 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:07.587303 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:07.587931 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:07.588109 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:07.589354 (+ 1245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:09:07.589360 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:09:07.589364 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:09:07.590058 (+ 694us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:09:07.590128 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:09:07.590166 (+ 38us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:09:07.590257 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 15:09:07.590435 (+ 178us) server_negotiation.cc:299] Negotiation successful
0926 15:09:07.590522 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 15:09:07.622004 15465 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 15:09:07.623819 15465 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:09:07.646023 15465 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f5fe0e3a202c4384b01c4ef139842702: Generated new TSK 0
I20260926 15:09:07.647048 15465 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 15:09:07.656056 12427 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 15376
I20260926 15:09:07.657946 15438 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:09:07.753783 12427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8uoJGv/build/asan/bin/kudu with pid 15376
2026-09-26T15:09:07Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (2012 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435318278681-12427-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 15:09:07 dist-test-slave-865v krb5kdc[15478](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 15:09:07 dist-test-slave-865v krb5kdc[15478](info): set up 2 sockets
Sep 26 15:09:07 dist-test-slave-865v krb5kdc[15478](info): commencing operation
krb5kdc: starting...
W20260926 15:09:08.857378 12427 mini_kdc.cc:121] Time spent starting KDC: real 1.068s user 0.000s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435318278681-12427-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435318278681-12427-0/krb5kdc/test-user.keytab.
Sep 26 15:09:08 dist-test-slave-865v krb5kdc[15478](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435348, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T15:09:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T15:09:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.34.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.34.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.34.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.254.keytab.
Entry for principal kudu/127.12.34.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.254.keytab.
I20260926 15:09:09.045349 12427 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8uoJGv/build/asan/bin/kudu
/tmp/dist-test-task8uoJGv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435318278681-12427-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435318278681-12427-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435318278681-12427-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.34.254:41451
--webserver_interface=127.12.34.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.254.keytab
--principal=kudu/127.12.34.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.34.212:41583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.34.254:41451
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435318278681-12427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435318278681-12427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435318278681-12427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:09:09.357748 15491 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:09:09.358028 15491 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:09:09.358116 15491 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:09:09.365406 15491 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:09:09.365527 15491 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:09:09.365574 15491 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:09:09.365622 15491 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:09:09.365664 15491 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:09:09.375492 15491 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.34.212:41583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435318278681-12427-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435318278681-12427-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.34.254:41451
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.254.keytab
--principal=kudu/127.12.34.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.34.254:41451
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.34.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435318278681-12427-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:09:09.376768 15491 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:09:09.378015 15491 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:09:09.390203 15496 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:09:09.390316 15497 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:09:09.390196 15499 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:09:09.391415 15491 server_base.cc:1034] running on GCE node
I20260926 15:09:09.392128 15491 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:09:09.393576 15491 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:09:09.394793 15491 hybrid_clock.cc:648] HybridClock initialized: now 1790435349394748 us; error 33 us; skew 500 ppm
I20260926 15:09:09.395157 15491 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:09:09 dist-test-slave-865v krb5kdc[15478](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435349, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:09:09.398373 15491 init.cc:377] Logged in from keytab as kudu/127.12.34.254@KRBTEST.COM (short username kudu)
I20260926 15:09:09.399864 15491 webserver.cc:460] Webserver started at http://127.12.34.254:32915/ using document root <none> and password file <none>
I20260926 15:09:09.400480 15491 fs_manager.cc:359] Metadata directory not provided
I20260926 15:09:09.400579 15491 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:09:09.400825 15491 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:09:09.403841 15491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435318278681-12427-0/minicluster-data/master-0/data/instance:
uuid: "1243431b99a74e61baeb6cbceb34ba93"
format_stamp: "Formatted at 2026-09-26 15:09:09 on dist-test-slave-865v"
server_key: "5b76d7cf351cfe6af289595fb4c4eaa6"
server_key_iv: "948c7cf9df2573d524df2b9d9028738f"
server_key_version: "encryptionkey@0"
I20260926 15:09:09.404533 15491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435318278681-12427-0/minicluster-data/master-0/wal/instance:
uuid: "1243431b99a74e61baeb6cbceb34ba93"
format_stamp: "Formatted at 2026-09-26 15:09:09 on dist-test-slave-865v"
server_key: "5b76d7cf351cfe6af289595fb4c4eaa6"
server_key_iv: "948c7cf9df2573d524df2b9d9028738f"
server_key_version: "encryptionkey@0"
I20260926 15:09:09.410332 15491 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260926 15:09:09.414737 15506 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:09:09.416597 15491 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260926 15:09:09.417215 15491 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435318278681-12427-0/minicluster-data/master-0/data,/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435318278681-12427-0/minicluster-data/master-0/wal
uuid: "1243431b99a74e61baeb6cbceb34ba93"
format_stamp: "Formatted at 2026-09-26 15:09:09 on dist-test-slave-865v"
server_key: "5b76d7cf351cfe6af289595fb4c4eaa6"
server_key_iv: "948c7cf9df2573d524df2b9d9028738f"
server_key_version: "encryptionkey@0"
I20260926 15:09:09.417546 15491 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435318278681-12427-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435318278681-12427-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435318278681-12427-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:09:09.437659 15491 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:09:09.442942 15491 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:09:09.443202 15491 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:09:09.463081 15491 rpc_server.cc:307] RPC server started. Bound to: 127.12.34.254:41451
I20260926 15:09:09.463102 15557 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.34.254:41451 every 8 connection(s)
I20260926 15:09:09.464541 15491 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
I20260926 15:09:09.468127 15558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:09:09.473027 12427 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8uoJGv/build/asan/bin/kudu as pid 15491
I20260926 15:09:09.473238 12427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435318278681-12427-0/minicluster-data/master-0/wal/instance
I20260926 15:09:09.473994 12427 external_mini_cluster.cc:1428] Setting key 715cfde51f36d440d8a373759eeec08c
I20260926 15:09:09.479632 15558 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1243431b99a74e61baeb6cbceb34ba93: Bootstrap starting.
I20260926 15:09:09.483114 15558 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1243431b99a74e61baeb6cbceb34ba93: Neither blocks nor log segments found. Creating new log.
I20260926 15:09:09.484288 15558 log.cc:824] T 00000000000000000000000000000000 P 1243431b99a74e61baeb6cbceb34ba93: Log is configured to *not* fsync() on all Append() calls
I20260926 15:09:09.487803 15558 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1243431b99a74e61baeb6cbceb34ba93: No bootstrap required, opened a new log
Sep 26 15:09:09 dist-test-slave-865v krb5kdc[15478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435348, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.34.254@KRBTEST.COM
I20260926 15:09:09.495568 15558 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1243431b99a74e61baeb6cbceb34ba93 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1243431b99a74e61baeb6cbceb34ba93" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 41451 } }
I20260926 15:09:09.495937 15558 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1243431b99a74e61baeb6cbceb34ba93 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:09:09.496061 15558 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1243431b99a74e61baeb6cbceb34ba93 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1243431b99a74e61baeb6cbceb34ba93, State: Initialized, Role: FOLLOWER
I20260926 15:09:09.496610 15558 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1243431b99a74e61baeb6cbceb34ba93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1243431b99a74e61baeb6cbceb34ba93" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 41451 } }
I20260926 15:09:09.496851 15558 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1243431b99a74e61baeb6cbceb34ba93 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 15:09:09.497025 15558 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1243431b99a74e61baeb6cbceb34ba93 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 15:09:09.497225 15558 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1243431b99a74e61baeb6cbceb34ba93 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:09:09.498986 15561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:09.476045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47378 (local address 127.12.34.254:41451)
0926 15:09:09.476914 (+ 869us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:09.476932 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:09.476986 (+ 54us) server_negotiation.cc:407] Connection header received
0926 15:09:09.478987 (+ 2001us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:09.479009 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:09.479368 (+ 359us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:09.479688 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:09:09.481328 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:09.482622 (+ 1294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:09.484473 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:09.485514 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:09.485710 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:09.490374 (+ 4664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:09:09.490381 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:09:09.490397 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:09:09.490437 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:09:09.494286 (+ 3849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:09.495097 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:09.495104 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:09.495111 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:09.495334 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:09.495813 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:09.495818 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:09.495822 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:09.496627 (+ 805us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:09:09.496958 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0926 15:09:09.497468 (+ 510us) server_negotiation.cc:299] Negotiation successful
0926 15:09:09.497932 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":616,"thread_start_us":411,"threads_started":1}
I20260926 15:09:09.499696 15558 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1243431b99a74e61baeb6cbceb34ba93 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1243431b99a74e61baeb6cbceb34ba93" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 41451 } }
I20260926 15:09:09.500211 15558 leader_election.cc:302] T 00000000000000000000000000000000 P 1243431b99a74e61baeb6cbceb34ba93 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1243431b99a74e61baeb6cbceb34ba93; no voters:
I20260926 15:09:09.500845 15558 leader_election.cc:288] T 00000000000000000000000000000000 P 1243431b99a74e61baeb6cbceb34ba93 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 15:09:09.501040 15563 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1243431b99a74e61baeb6cbceb34ba93 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:09:09.501735 15563 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1243431b99a74e61baeb6cbceb34ba93 [term 1 LEADER]: Becoming Leader. State: Replica: 1243431b99a74e61baeb6cbceb34ba93, State: Running, Role: LEADER
I20260926 15:09:09.502687 15563 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1243431b99a74e61baeb6cbceb34ba93 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1243431b99a74e61baeb6cbceb34ba93" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 41451 } }
I20260926 15:09:09.504370 15563 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1243431b99a74e61baeb6cbceb34ba93 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1243431b99a74e61baeb6cbceb34ba93. Latest consensus state: current_term: 1 leader_uuid: "1243431b99a74e61baeb6cbceb34ba93" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1243431b99a74e61baeb6cbceb34ba93" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 41451 } } }
I20260926 15:09:09.504724 15563 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1243431b99a74e61baeb6cbceb34ba93 [sys.catalog]: This master's current role is: LEADER
I20260926 15:09:09.505302 15565 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1243431b99a74e61baeb6cbceb34ba93 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1243431b99a74e61baeb6cbceb34ba93" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1243431b99a74e61baeb6cbceb34ba93" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 41451 } } }
I20260926 15:09:09.505852 15565 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1243431b99a74e61baeb6cbceb34ba93 [sys.catalog]: This master's current role is: LEADER
I20260926 15:09:09.508476 15558 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1243431b99a74e61baeb6cbceb34ba93 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:09:09.508668 15567 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:09:09.514027 15567 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 15:09:09.525560 15567 catalog_manager.cc:1347] Generated new cluster ID: be466c577615449993a413a4fab9431f
I20260926 15:09:09.525695 15567 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 15:09:09.542083 15567 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 15:09:09.543496 15567 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:09:09.557711 15567 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1243431b99a74e61baeb6cbceb34ba93: Generated new TSK 0
I20260926 15:09:09.558548 15567 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.34.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.34.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.34.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.193.keytab.
Entry for principal kudu/127.12.34.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.193.keytab.
I20260926 15:09:09.602630 12427 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8uoJGv/build/asan/bin/kudu
/tmp/dist-test-task8uoJGv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435318278681-12427-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435318278681-12427-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435318278681-12427-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435318278681-12427-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.34.193:0
--local_ip_for_outbound_sockets=127.12.34.193
--webserver_interface=127.12.34.193
--webserver_port=0
--tserver_master_addrs=127.12.34.254:41451
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.193.keytab
--principal=kudu/127.12.34.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.34.212:41583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435318278681-12427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435318278681-12427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435318278681-12427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:09:09.937204 15584 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:09:09.937512 15584 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:09:09.937668 15584 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:09:09.945742 15584 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:09:09.945873 15584 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:09:09.946072 15584 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.34.193
I20260926 15:09:09.956709 15584 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.34.212:41583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435318278681-12427-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435318278681-12427-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.193.keytab
--principal=kudu/127.12.34.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.34.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435318278681-12427-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.34.193
--webserver_port=0
--tserver_master_addrs=127.12.34.254:41451
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.34.193
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435318278681-12427-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:09:09.958266 15584 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:09:09.959615 15584 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:09:09.972348 15589 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:09:09.972389 15590 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:09:09.972348 15592 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:09:09.973318 15584 server_base.cc:1034] running on GCE node
I20260926 15:09:09.973891 15584 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:09:09.974776 15584 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:09:09.976027 15584 hybrid_clock.cc:648] HybridClock initialized: now 1790435349975983 us; error 67 us; skew 500 ppm
I20260926 15:09:09.976388 15584 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:09:09 dist-test-slave-865v krb5kdc[15478](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435349, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:09:09.979669 15584 init.cc:377] Logged in from keytab as kudu/127.12.34.193@KRBTEST.COM (short username kudu)
I20260926 15:09:09.981228 15584 webserver.cc:460] Webserver started at http://127.12.34.193:38269/ using document root <none> and password file <none>
I20260926 15:09:09.981913 15584 fs_manager.cc:359] Metadata directory not provided
I20260926 15:09:09.982046 15584 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:09:09.982398 15584 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:09:09.985597 15584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435318278681-12427-0/minicluster-data/ts-0/data/instance:
uuid: "ff3e8727ca6b48f69c3e92ff71955229"
format_stamp: "Formatted at 2026-09-26 15:09:09 on dist-test-slave-865v"
server_key: "7b0642e3b0bbaf2db3c1687dee28fbe4"
server_key_iv: "e207419fb9d42055fb66d710bdb2c78f"
server_key_version: "encryptionkey@0"
I20260926 15:09:09.986425 15584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435318278681-12427-0/minicluster-data/ts-0/wal/instance:
uuid: "ff3e8727ca6b48f69c3e92ff71955229"
format_stamp: "Formatted at 2026-09-26 15:09:09 on dist-test-slave-865v"
server_key: "7b0642e3b0bbaf2db3c1687dee28fbe4"
server_key_iv: "e207419fb9d42055fb66d710bdb2c78f"
server_key_version: "encryptionkey@0"
I20260926 15:09:09.991672 15584 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260926 15:09:09.995144 15599 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:09:09.996423 15584 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 15:09:09.996592 15584 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435318278681-12427-0/minicluster-data/ts-0/data,/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435318278681-12427-0/minicluster-data/ts-0/wal
uuid: "ff3e8727ca6b48f69c3e92ff71955229"
format_stamp: "Formatted at 2026-09-26 15:09:09 on dist-test-slave-865v"
server_key: "7b0642e3b0bbaf2db3c1687dee28fbe4"
server_key_iv: "e207419fb9d42055fb66d710bdb2c78f"
server_key_version: "encryptionkey@0"
I20260926 15:09:09.996773 15584 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435318278681-12427-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435318278681-12427-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435318278681-12427-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:09:10.019250 15584 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:09:10.024356 15584 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:09:10.024670 15584 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:09:10.025803 15584 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:09:10.027199 15584 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:09:10.027321 15584 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:09:10.027472 15584 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:09:10.027544 15584 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:09:10.066471 15584 rpc_server.cc:307] RPC server started. Bound to: 127.12.34.193:39501
I20260926 15:09:10.066488 15711 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.34.193:39501 every 8 connection(s)
I20260926 15:09:10.068346 15584 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435318278681-12427-0/minicluster-data/ts-0/data/info.pb
I20260926 15:09:10.071957 12427 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8uoJGv/build/asan/bin/kudu as pid 15584
I20260926 15:09:10.072190 12427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435318278681-12427-0/minicluster-data/ts-0/wal/instance
I20260926 15:09:10.073166 12427 external_mini_cluster.cc:1428] Setting key 512c68c99a91850799eb4257c402d1ce
Sep 26 15:09:10 dist-test-slave-865v krb5kdc[15478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435349, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.193@KRBTEST.COM for kudu/127.12.34.254@KRBTEST.COM
I20260926 15:09:10.090858 15714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:10.071489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.193:41067 (local address 127.12.34.254:41451)
0926 15:09:10.072139 (+ 650us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:10.072149 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:10.074347 (+ 2198us) server_negotiation.cc:407] Connection header received
0926 15:09:10.075385 (+ 1038us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:10.075390 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:10.075486 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:10.075612 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:09:10.077134 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:10.077857 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:10.079399 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:10.079796 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:10.079930 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:10.083909 (+ 3979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:09:10.083915 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:09:10.083918 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:09:10.083953 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:09:10.087655 (+ 3702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:10.088461 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:10.088467 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:10.088472 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:10.088596 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:10.089023 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:10.089030 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:10.089033 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:10.089361 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:09:10.089525 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0926 15:09:10.090212 (+ 687us) server_negotiation.cc:299] Negotiation successful
0926 15:09:10.090472 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":483,"thread_start_us":343,"threads_started":1}
I20260926 15:09:10.092168 15715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:10.072212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:41451 (local address 127.12.34.193:41067)
0926 15:09:10.073175 (+ 963us) negotiation.cc:107] Waiting for socket to connect
0926 15:09:10.073222 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 15:09:10.075111 (+ 1889us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:09:10.075853 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:09:10.075862 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:09:10.076368 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:09:10.076939 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:10.076960 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:10.078025 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:10.078030 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:10.079245 (+ 1215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:10.079256 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:10.080042 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:10.080049 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:10.080273 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:10.081068 (+ 795us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:09:10.081253 (+ 185us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:09:10.083648 (+ 2395us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:09:10.087841 (+ 4193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:09:10.087851 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:09:10.087876 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:09:10.088326 (+ 450us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:09:10.088711 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:09:10.088717 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:09:10.088720 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:09:10.088906 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:09:10.089500 (+ 594us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:09:10.089509 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:09:10.089855 (+ 346us) client_negotiation.cc:769] Sending connection context
0926 15:09:10.090132 (+ 277us) client_negotiation.cc:240] Negotiation successful
0926 15:09:10.090685 (+ 553us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":624,"thread_start_us":400,"threads_started":1}
WARNING: no policy specified for kudu/127.12.34.194@KRBTEST.COM; defaulting to no policy
I20260926 15:09:10.094358 15712 heartbeater.cc:347] Connected to a master server at 127.12.34.254:41451
I20260926 15:09:10.094911 15712 heartbeater.cc:464] Registering TS with master...
I20260926 15:09:10.095657 15712 heartbeater.cc:511] Master 127.12.34.254:41451 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.34.194@KRBTEST.COM" created.
I20260926 15:09:10.098438 15523 ts_manager.cc:194] Registered new tserver with Master: ff3e8727ca6b48f69c3e92ff71955229 (127.12.34.193:39501)
I20260926 15:09:10.100044 15523 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.34.193@KRBTEST.COM'} at 127.12.34.193:41067
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.34.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.194.keytab.
Entry for principal kudu/127.12.34.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.194.keytab.
I20260926 15:09:10.128736 12427 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8uoJGv/build/asan/bin/kudu
/tmp/dist-test-task8uoJGv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435318278681-12427-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435318278681-12427-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435318278681-12427-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435318278681-12427-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.34.194:0
--local_ip_for_outbound_sockets=127.12.34.194
--webserver_interface=127.12.34.194
--webserver_port=0
--tserver_master_addrs=127.12.34.254:41451
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.194.keytab
--principal=kudu/127.12.34.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.34.212:41583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435318278681-12427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435318278681-12427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435318278681-12427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:09:10.455879 15718 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:09:10.456173 15718 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:09:10.456362 15718 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:09:10.463403 15718 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:09:10.463546 15718 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:09:10.463717 15718 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.34.194
I20260926 15:09:10.473050 15718 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.34.212:41583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435318278681-12427-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435318278681-12427-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.194.keytab
--principal=kudu/127.12.34.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.34.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435318278681-12427-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.34.194
--webserver_port=0
--tserver_master_addrs=127.12.34.254:41451
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.34.194
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435318278681-12427-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:09:10.474298 15718 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:09:10.475548 15718 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:09:10.488765 15723 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:09:10.488799 15726 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:09:10.488822 15724 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:09:10.489615 15718 server_base.cc:1034] running on GCE node
I20260926 15:09:10.490187 15718 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:09:10.491125 15718 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:09:10.492396 15718 hybrid_clock.cc:648] HybridClock initialized: now 1790435350492332 us; error 59 us; skew 500 ppm
I20260926 15:09:10.492754 15718 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:09:10 dist-test-slave-865v krb5kdc[15478](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435350, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:09:10.496275 15718 init.cc:377] Logged in from keytab as kudu/127.12.34.194@KRBTEST.COM (short username kudu)
I20260926 15:09:10.497902 15718 webserver.cc:460] Webserver started at http://127.12.34.194:36709/ using document root <none> and password file <none>
I20260926 15:09:10.498605 15718 fs_manager.cc:359] Metadata directory not provided
I20260926 15:09:10.498749 15718 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:09:10.499049 15718 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:09:10.502041 15718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435318278681-12427-0/minicluster-data/ts-1/data/instance:
uuid: "5c1b0d7b1dbf4e7796eae80bd774c129"
format_stamp: "Formatted at 2026-09-26 15:09:10 on dist-test-slave-865v"
server_key: "35015a0fe357365bb1bb0ad824535709"
server_key_iv: "be8c75c88b1b506c00ad3d557ccc403d"
server_key_version: "encryptionkey@0"
I20260926 15:09:10.502797 15718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435318278681-12427-0/minicluster-data/ts-1/wal/instance:
uuid: "5c1b0d7b1dbf4e7796eae80bd774c129"
format_stamp: "Formatted at 2026-09-26 15:09:10 on dist-test-slave-865v"
server_key: "35015a0fe357365bb1bb0ad824535709"
server_key_iv: "be8c75c88b1b506c00ad3d557ccc403d"
server_key_version: "encryptionkey@0"
I20260926 15:09:10.508081 15718 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 15:09:10.512219 15733 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:09:10.513643 15718 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 15:09:10.513988 15718 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435318278681-12427-0/minicluster-data/ts-1/data,/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435318278681-12427-0/minicluster-data/ts-1/wal
uuid: "5c1b0d7b1dbf4e7796eae80bd774c129"
format_stamp: "Formatted at 2026-09-26 15:09:10 on dist-test-slave-865v"
server_key: "35015a0fe357365bb1bb0ad824535709"
server_key_iv: "be8c75c88b1b506c00ad3d557ccc403d"
server_key_version: "encryptionkey@0"
I20260926 15:09:10.514411 15718 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435318278681-12427-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435318278681-12427-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435318278681-12427-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:09:10.533933 15718 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:09:10.539198 15718 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:09:10.539543 15718 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:09:10.540503 15718 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:09:10.541985 15718 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:09:10.542090 15718 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:09:10.542238 15718 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:09:10.542309 15718 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:09:10.583544 15718 rpc_server.cc:307] RPC server started. Bound to: 127.12.34.194:42303
I20260926 15:09:10.583586 15845 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.34.194:42303 every 8 connection(s)
I20260926 15:09:10.584914 15718 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435318278681-12427-0/minicluster-data/ts-1/data/info.pb
I20260926 15:09:10.589674 12427 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8uoJGv/build/asan/bin/kudu as pid 15718
I20260926 15:09:10.590066 12427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435318278681-12427-0/minicluster-data/ts-1/wal/instance
I20260926 15:09:10.590715 12427 external_mini_cluster.cc:1428] Setting key 1f2b7025c97d1c719b9120f20e797d23
Sep 26 15:09:10 dist-test-slave-865v krb5kdc[15478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435350, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.194@KRBTEST.COM for kudu/127.12.34.254@KRBTEST.COM
I20260926 15:09:10.609863 15714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:10.587769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.194:39277 (local address 127.12.34.254:41451)
0926 15:09:10.587953 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:10.587958 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:10.589796 (+ 1838us) server_negotiation.cc:407] Connection header received
0926 15:09:10.592661 (+ 2865us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:10.592669 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:10.592775 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:10.592898 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:09:10.595110 (+ 2212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:10.596039 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:10.597804 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:10.598362 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:10.598492 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:10.603091 (+ 4599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:09:10.603096 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:09:10.603100 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:09:10.603132 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:09:10.605891 (+ 2759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:10.606894 (+ 1003us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:10.606898 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:10.606900 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:10.607032 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:10.607612 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:10.607615 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:10.607617 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:10.607873 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:09:10.608059 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0926 15:09:10.609059 (+ 1000us) server_negotiation.cc:299] Negotiation successful
0926 15:09:10.609389 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 15:09:10.610713 15848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:10.588184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:41451 (local address 127.12.34.194:39277)
0926 15:09:10.589627 (+ 1443us) negotiation.cc:107] Waiting for socket to connect
0926 15:09:10.589683 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0926 15:09:10.592321 (+ 2638us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:09:10.593262 (+ 941us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:09:10.593277 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:09:10.594037 (+ 760us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:09:10.594869 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:10.594895 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:10.596202 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:10.596208 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:10.597602 (+ 1394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:10.597634 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:10.598580 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:10.598587 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:10.598859 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:10.599664 (+ 805us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:09:10.599686 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:09:10.602818 (+ 3132us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:09:10.606031 (+ 3213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:09:10.606043 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:09:10.606063 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:09:10.606742 (+ 679us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:09:10.607160 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:09:10.607169 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:09:10.607176 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:09:10.607478 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:09:10.608020 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:09:10.608031 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:09:10.608562 (+ 531us) client_negotiation.cc:769] Sending connection context
0926 15:09:10.608945 (+ 383us) client_negotiation.cc:240] Negotiation successful
0926 15:09:10.609506 (+ 561us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1108,"thread_start_us":737,"threads_started":1}
I20260926 15:09:10.612123 15846 heartbeater.cc:347] Connected to a master server at 127.12.34.254:41451
WARNING: no policy specified for kudu/127.12.34.195@KRBTEST.COM; defaulting to no policy
I20260926 15:09:10.612521 15846 heartbeater.cc:464] Registering TS with master...
I20260926 15:09:10.613350 15846 heartbeater.cc:511] Master 127.12.34.254:41451 requested a full tablet report, sending...
I20260926 15:09:10.614801 15523 ts_manager.cc:194] Registered new tserver with Master: 5c1b0d7b1dbf4e7796eae80bd774c129 (127.12.34.194:42303)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.34.195@KRBTEST.COM" created.
I20260926 15:09:10.615979 15523 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.34.194@KRBTEST.COM'} at 127.12.34.194:39277
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.34.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.195.keytab.
Entry for principal kudu/127.12.34.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.195.keytab.
I20260926 15:09:10.640054 12427 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8uoJGv/build/asan/bin/kudu
/tmp/dist-test-task8uoJGv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435318278681-12427-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435318278681-12427-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435318278681-12427-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435318278681-12427-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.34.195:0
--local_ip_for_outbound_sockets=127.12.34.195
--webserver_interface=127.12.34.195
--webserver_port=0
--tserver_master_addrs=127.12.34.254:41451
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.195.keytab
--principal=kudu/127.12.34.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.34.212:41583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435318278681-12427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435318278681-12427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435318278681-12427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:09:10.976313 15851 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:09:10.976624 15851 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:09:10.976799 15851 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:09:10.984026 15851 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:09:10.984179 15851 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:09:10.984359 15851 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.34.195
I20260926 15:09:10.993988 15851 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.34.212:41583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435318278681-12427-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435318278681-12427-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.195.keytab
--principal=kudu/127.12.34.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.34.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435318278681-12427-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.34.195
--webserver_port=0
--tserver_master_addrs=127.12.34.254:41451
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.34.195
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435318278681-12427-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:09:10.995194 15851 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:09:10.996423 15851 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:09:11.009295 15857 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:09:11.009222 15859 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:09:11.009369 15856 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:09:11.010233 15851 server_base.cc:1034] running on GCE node
I20260926 15:09:11.010844 15851 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:09:11.011921 15851 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:09:11.013217 15851 hybrid_clock.cc:648] HybridClock initialized: now 1790435351013154 us; error 49 us; skew 500 ppm
I20260926 15:09:11.013549 15851 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:09:11 dist-test-slave-865v krb5kdc[15478](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435351, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:09:11.016649 15851 init.cc:377] Logged in from keytab as kudu/127.12.34.195@KRBTEST.COM (short username kudu)
I20260926 15:09:11.018090 15851 webserver.cc:460] Webserver started at http://127.12.34.195:40889/ using document root <none> and password file <none>
I20260926 15:09:11.018833 15851 fs_manager.cc:359] Metadata directory not provided
I20260926 15:09:11.018934 15851 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:09:11.019279 15851 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:09:11.022027 15851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435318278681-12427-0/minicluster-data/ts-2/data/instance:
uuid: "d15c2dcf1dea4c288b75f190a95a59cc"
format_stamp: "Formatted at 2026-09-26 15:09:11 on dist-test-slave-865v"
server_key: "d2deb933c182e11c3c160128c33308a3"
server_key_iv: "44f168af4aacf15d0d71e6e09f7afd64"
server_key_version: "encryptionkey@0"
I20260926 15:09:11.022822 15851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435318278681-12427-0/minicluster-data/ts-2/wal/instance:
uuid: "d15c2dcf1dea4c288b75f190a95a59cc"
format_stamp: "Formatted at 2026-09-26 15:09:11 on dist-test-slave-865v"
server_key: "d2deb933c182e11c3c160128c33308a3"
server_key_iv: "44f168af4aacf15d0d71e6e09f7afd64"
server_key_version: "encryptionkey@0"
I20260926 15:09:11.027822 15851 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 15:09:11.031139 15866 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:09:11.032407 15851 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 15:09:11.032601 15851 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435318278681-12427-0/minicluster-data/ts-2/data,/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435318278681-12427-0/minicluster-data/ts-2/wal
uuid: "d15c2dcf1dea4c288b75f190a95a59cc"
format_stamp: "Formatted at 2026-09-26 15:09:11 on dist-test-slave-865v"
server_key: "d2deb933c182e11c3c160128c33308a3"
server_key_iv: "44f168af4aacf15d0d71e6e09f7afd64"
server_key_version: "encryptionkey@0"
I20260926 15:09:11.032787 15851 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435318278681-12427-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435318278681-12427-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435318278681-12427-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:09:11.074864 15851 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:09:11.079078 15851 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:09:11.079411 15851 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:09:11.080403 15851 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:09:11.081784 15851 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:09:11.081887 15851 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:09:11.082057 15851 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:09:11.082129 15851 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:09:11.105230 15712 heartbeater.cc:503] Master 127.12.34.254:41451 was elected leader, sending a full tablet report...
I20260926 15:09:11.116710 15851 rpc_server.cc:307] RPC server started. Bound to: 127.12.34.195:37201
I20260926 15:09:11.116753 15978 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.34.195:37201 every 8 connection(s)
I20260926 15:09:11.118018 15851 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435318278681-12427-0/minicluster-data/ts-2/data/info.pb
I20260926 15:09:11.118355 12427 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8uoJGv/build/asan/bin/kudu as pid 15851
I20260926 15:09:11.118528 12427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790435318278681-12427-0/minicluster-data/ts-2/wal/instance
I20260926 15:09:11.119093 12427 external_mini_cluster.cc:1428] Setting key f8f49319eba8cb36163c2b02e9192289
Sep 26 15:09:11 dist-test-slave-865v krb5kdc[15478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435351, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.195@KRBTEST.COM for kudu/127.12.34.254@KRBTEST.COM
I20260926 15:09:11.138331 15981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:11.121293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.195:51205 (local address 127.12.34.254:41451)
0926 15:09:11.121897 (+ 604us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:11.121903 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:11.122981 (+ 1078us) server_negotiation.cc:407] Connection header received
0926 15:09:11.124778 (+ 1797us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:11.124782 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:11.124845 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:11.124938 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:09:11.126422 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:11.127211 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:11.128644 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:11.129035 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:11.129248 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:11.132555 (+ 3307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:09:11.132560 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:09:11.132563 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:09:11.132593 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:09:11.135497 (+ 2904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:11.136253 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:11.136257 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:11.136260 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:11.136370 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:11.136721 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:11.136724 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:11.136726 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:11.137022 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:09:11.137187 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0926 15:09:11.137923 (+ 736us) server_negotiation.cc:299] Negotiation successful
0926 15:09:11.138073 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":449,"thread_start_us":326,"threads_started":1}
I20260926 15:09:11.139318 15982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:11.121962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:41451 (local address 127.12.34.195:51205)
0926 15:09:11.122835 (+ 873us) negotiation.cc:107] Waiting for socket to connect
0926 15:09:11.122888 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 15:09:11.124538 (+ 1650us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:09:11.125247 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:09:11.125255 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:09:11.125747 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:09:11.126269 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:11.126284 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:11.127351 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:11.127355 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:11.128516 (+ 1161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:11.128527 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:11.129319 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:11.129323 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:11.129500 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:11.130176 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:09:11.130197 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:09:11.132302 (+ 2105us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:09:11.135612 (+ 3310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:09:11.135618 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:09:11.135630 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:09:11.136146 (+ 516us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:09:11.136461 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:09:11.136464 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:09:11.136467 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:09:11.136633 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:09:11.137174 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:09:11.137189 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:09:11.137608 (+ 419us) client_negotiation.cc:769] Sending connection context
0926 15:09:11.137838 (+ 230us) client_negotiation.cc:240] Negotiation successful
0926 15:09:11.138200 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":411,"threads_started":1}
I20260926 15:09:11.140568 15979 heartbeater.cc:347] Connected to a master server at 127.12.34.254:41451
I20260926 15:09:11.140820 15979 heartbeater.cc:464] Registering TS with master...
I20260926 15:09:11.141533 15979 heartbeater.cc:511] Master 127.12.34.254:41451 requested a full tablet report, sending...
I20260926 15:09:11.142866 15523 ts_manager.cc:194] Registered new tserver with Master: d15c2dcf1dea4c288b75f190a95a59cc (127.12.34.195:37201)
I20260926 15:09:11.143669 15523 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.34.195@KRBTEST.COM'} at 127.12.34.195:51205
I20260926 15:09:11.147164 12427 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 15:09:11 dist-test-slave-865v krb5kdc[15478](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435351, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 26 15:09:11 dist-test-slave-865v krb5kdc[15478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435351, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.12.34.254@KRBTEST.COM
I20260926 15:09:11.194968 15981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:11.179951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47388 (local address 127.12.34.254:41451)
0926 15:09:11.180142 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:11.180146 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:11.180604 (+ 458us) server_negotiation.cc:407] Connection header received
0926 15:09:11.180747 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:11.180752 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:11.180839 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:11.180955 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:09:11.182834 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:11.183573 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:11.184791 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:11.185266 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:11.185361 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:11.189812 (+ 4451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:09:11.189817 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:09:11.189821 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:09:11.189857 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:09:11.192557 (+ 2700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:11.193234 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:11.193238 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:11.193241 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:11.193373 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:11.193755 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:11.193759 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:11.193761 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:11.194088 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:09:11.194278 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0926 15:09:11.194573 (+ 295us) server_negotiation.cc:299] Negotiation successful
0926 15:09:11.194736 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
W20260926 15:09:11.195600 15523 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:47388
I20260926 15:09:11.197393 12427 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 15584
Sep 26 15:09:11 dist-test-slave-865v krb5kdc[15478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435351, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.12.34.193@KRBTEST.COM
I20260926 15:09:11.213531 15991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:11.198537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47016 (local address 127.12.34.193:39501)
0926 15:09:11.199189 (+ 652us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:11.199199 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:11.199235 (+ 36us) server_negotiation.cc:407] Connection header received
0926 15:09:11.199290 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:11.199299 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:11.199507 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:11.199675 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:09:11.200976 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:11.202262 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:11.203461 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:11.204012 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:11.204154 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:11.208044 (+ 3890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:09:11.208055 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:09:11.208068 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:09:11.208139 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:09:11.210939 (+ 2800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:11.211587 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:11.211596 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:11.211604 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:11.211714 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:11.212065 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:11.212071 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:11.212074 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:11.212387 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:09:11.212678 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0926 15:09:11.212911 (+ 233us) server_negotiation.cc:299] Negotiation successful
0926 15:09:11.213188 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":463,"thread_start_us":279,"threads_started":1}
W20260926 15:09:11.214656 15707 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:47016
W20260926 15:09:11.215626 12427 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 15584: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 15:09:11.215786 12427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8uoJGv/build/asan/bin/kudu with pid 15584
I20260926 15:09:11.231840 12427 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 15718
Sep 26 15:09:11 dist-test-slave-865v krb5kdc[15478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435351, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.12.34.194@KRBTEST.COM
I20260926 15:09:11.248199 15992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:11.232709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35566 (local address 127.12.34.194:42303)
0926 15:09:11.233276 (+ 567us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:11.233283 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:11.233306 (+ 23us) server_negotiation.cc:407] Connection header received
0926 15:09:11.233370 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:11.233375 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:11.233567 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:11.233694 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:09:11.234788 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:11.235742 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:11.236962 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:11.237717 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:11.237838 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:11.242021 (+ 4183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:09:11.242030 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:09:11.242045 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:09:11.242077 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:09:11.245588 (+ 3511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:11.246263 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:11.246270 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:11.246272 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:11.246392 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:11.246856 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:11.246866 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:11.246868 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:11.247203 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:09:11.247433 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0926 15:09:11.247716 (+ 283us) server_negotiation.cc:299] Negotiation successful
0926 15:09:11.247907 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":428,"thread_start_us":315,"threads_started":1}
W20260926 15:09:11.249583 15841 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:35566
W20260926 15:09:11.250761 12427 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 15718: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 15:09:11.250895 12427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8uoJGv/build/asan/bin/kudu with pid 15718
I20260926 15:09:11.266006 12427 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 15851
Sep 26 15:09:11 dist-test-slave-865v krb5kdc[15478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435351, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.12.34.195@KRBTEST.COM
I20260926 15:09:11.282295 15993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:11.267087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43712 (local address 127.12.34.195:37201)
0926 15:09:11.267704 (+ 617us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:11.267710 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:11.267738 (+ 28us) server_negotiation.cc:407] Connection header received
0926 15:09:11.267791 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:11.267796 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:11.268016 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:11.268134 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:09:11.269475 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:11.270482 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:11.271711 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:11.272232 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:11.272325 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:11.276707 (+ 4382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:09:11.276718 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:09:11.276731 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:09:11.276760 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:09:11.279512 (+ 2752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:11.280094 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:11.280100 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:11.280102 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:11.280206 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:11.280835 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:11.280838 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:11.280840 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:11.281211 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:09:11.281435 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0926 15:09:11.281720 (+ 285us) server_negotiation.cc:299] Negotiation successful
0926 15:09:11.281899 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":483,"thread_start_us":338,"threads_started":1}
W20260926 15:09:11.283516 15974 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:43712
W20260926 15:09:11.284504 12427 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 15851: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 15:09:11.284651 12427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8uoJGv/build/asan/bin/kudu with pid 15851
I20260926 15:09:11.300081 12427 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 15491
I20260926 15:09:11.301260 15553 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:09:11.402469 12427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8uoJGv/build/asan/bin/kudu with pid 15491
2026-09-26T15:09:11Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3654 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435318278681-12427-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 15:09:11 dist-test-slave-865v krb5kdc[15997](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 15:09:11 dist-test-slave-865v krb5kdc[15997](info): set up 2 sockets
Sep 26 15:09:11 dist-test-slave-865v krb5kdc[15997](info): commencing operation
krb5kdc: starting...
W20260926 15:09:12.517410 12427 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.003s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435318278681-12427-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435318278681-12427-0/krb5kdc/test-user.keytab.
Sep 26 15:09:12 dist-test-slave-865v krb5kdc[15997](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435352, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T15:09:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T15:09:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.34.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.34.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.34.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.254.keytab.
Entry for principal kudu/127.12.34.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.254.keytab.
I20260926 15:09:12.709048 12427 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8uoJGv/build/asan/bin/kudu
/tmp/dist-test-task8uoJGv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435318278681-12427-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435318278681-12427-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435318278681-12427-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.34.254:34067
--webserver_interface=127.12.34.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.254.keytab
--principal=kudu/127.12.34.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.34.212:45563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.34.254:34067
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435318278681-12427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435318278681-12427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435318278681-12427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:09:13.025102 16010 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:09:13.025357 16010 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:09:13.025491 16010 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:09:13.032371 16010 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:09:13.032469 16010 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:09:13.032567 16010 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:09:13.032661 16010 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:09:13.032747 16010 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:09:13.041935 16010 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.34.212:45563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435318278681-12427-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435318278681-12427-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.34.254:34067
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.254.keytab
--principal=kudu/127.12.34.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.34.254:34067
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.34.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435318278681-12427-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:09:13.043116 16010 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:09:13.044322 16010 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:09:13.056538 16015 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:09:13.056579 16018 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:09:13.056649 16016 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:09:13.057727 16010 server_base.cc:1034] running on GCE node
I20260926 15:09:13.058418 16010 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:09:13.059712 16010 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:09:13.060963 16010 hybrid_clock.cc:648] HybridClock initialized: now 1790435353060886 us; error 49 us; skew 500 ppm
I20260926 15:09:13.061316 16010 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:09:13 dist-test-slave-865v krb5kdc[15997](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435353, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:09:13.070490 16010 init.cc:377] Logged in from keytab as kudu/127.12.34.254@KRBTEST.COM (short username kudu)
I20260926 15:09:13.072010 16010 webserver.cc:460] Webserver started at http://127.12.34.254:44523/ using document root <none> and password file <none>
I20260926 15:09:13.072671 16010 fs_manager.cc:359] Metadata directory not provided
I20260926 15:09:13.072774 16010 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:09:13.073050 16010 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:09:13.076102 16010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435318278681-12427-0/minicluster-data/master-0/data/instance:
uuid: "6a7e6bbc2e9e4e738466019e611cb499"
format_stamp: "Formatted at 2026-09-26 15:09:13 on dist-test-slave-865v"
server_key: "937d35d7d6b2cb1a3805053b7ae49821"
server_key_iv: "dd3a75f7c5016e42fbbf6bffd3569c57"
server_key_version: "encryptionkey@0"
I20260926 15:09:13.076855 16010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435318278681-12427-0/minicluster-data/master-0/wal/instance:
uuid: "6a7e6bbc2e9e4e738466019e611cb499"
format_stamp: "Formatted at 2026-09-26 15:09:13 on dist-test-slave-865v"
server_key: "937d35d7d6b2cb1a3805053b7ae49821"
server_key_iv: "dd3a75f7c5016e42fbbf6bffd3569c57"
server_key_version: "encryptionkey@0"
I20260926 15:09:13.083931 16010 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.002s
I20260926 15:09:13.089013 16025 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:09:13.090910 16010 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260926 15:09:13.091099 16010 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435318278681-12427-0/minicluster-data/master-0/data,/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435318278681-12427-0/minicluster-data/master-0/wal
uuid: "6a7e6bbc2e9e4e738466019e611cb499"
format_stamp: "Formatted at 2026-09-26 15:09:13 on dist-test-slave-865v"
server_key: "937d35d7d6b2cb1a3805053b7ae49821"
server_key_iv: "dd3a75f7c5016e42fbbf6bffd3569c57"
server_key_version: "encryptionkey@0"
I20260926 15:09:13.091343 16010 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435318278681-12427-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435318278681-12427-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435318278681-12427-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:09:13.108435 16010 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:09:13.112385 16010 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:09:13.112623 16010 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:09:13.131958 16010 rpc_server.cc:307] RPC server started. Bound to: 127.12.34.254:34067
I20260926 15:09:13.131995 16076 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.34.254:34067 every 8 connection(s)
I20260926 15:09:13.133467 16010 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
I20260926 15:09:13.136804 12427 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8uoJGv/build/asan/bin/kudu as pid 16010
I20260926 15:09:13.137106 12427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435318278681-12427-0/minicluster-data/master-0/wal/instance
I20260926 15:09:13.137259 16077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:09:13.137800 12427 external_mini_cluster.cc:1428] Setting key b9571ffdfc98e130122f2f1150ceb20b
I20260926 15:09:13.150652 16077 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a7e6bbc2e9e4e738466019e611cb499: Bootstrap starting.
Sep 26 15:09:13 dist-test-slave-865v krb5kdc[15997](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435352, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.34.254@KRBTEST.COM
I20260926 15:09:13.153911 16077 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6a7e6bbc2e9e4e738466019e611cb499: Neither blocks nor log segments found. Creating new log.
I20260926 15:09:13.155210 16077 log.cc:824] T 00000000000000000000000000000000 P 6a7e6bbc2e9e4e738466019e611cb499: Log is configured to *not* fsync() on all Append() calls
I20260926 15:09:13.158249 16077 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a7e6bbc2e9e4e738466019e611cb499: No bootstrap required, opened a new log
I20260926 15:09:13.160761 16080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:13.139688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44644 (local address 127.12.34.254:34067)
0926 15:09:13.140499 (+ 811us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:13.140514 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:13.140559 (+ 45us) server_negotiation.cc:407] Connection header received
0926 15:09:13.142178 (+ 1619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:13.142212 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:13.142672 (+ 460us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:13.143096 (+ 424us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:09:13.144438 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:13.146302 (+ 1864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:13.147657 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:13.148700 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:13.148875 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:13.152684 (+ 3809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:09:13.152694 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:09:13.152710 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:09:13.152754 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:09:13.156503 (+ 3749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:13.157151 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:13.157160 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:13.157169 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:13.157355 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:13.157739 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:13.157744 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:13.157747 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:13.158381 (+ 634us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:09:13.158655 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0926 15:09:13.159094 (+ 439us) server_negotiation.cc:299] Negotiation successful
0926 15:09:13.159469 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":580,"thread_start_us":322,"threads_started":1}
I20260926 15:09:13.166724 16077 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6a7e6bbc2e9e4e738466019e611cb499 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a7e6bbc2e9e4e738466019e611cb499" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 34067 } }
I20260926 15:09:13.167048 16077 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6a7e6bbc2e9e4e738466019e611cb499 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:09:13.167161 16077 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6a7e6bbc2e9e4e738466019e611cb499 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a7e6bbc2e9e4e738466019e611cb499, State: Initialized, Role: FOLLOWER
I20260926 15:09:13.167733 16077 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6a7e6bbc2e9e4e738466019e611cb499 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a7e6bbc2e9e4e738466019e611cb499" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 34067 } }
I20260926 15:09:13.167964 16077 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6a7e6bbc2e9e4e738466019e611cb499 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 15:09:13.168102 16077 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6a7e6bbc2e9e4e738466019e611cb499 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 15:09:13.168273 16077 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6a7e6bbc2e9e4e738466019e611cb499 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:09:13.170348 16077 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6a7e6bbc2e9e4e738466019e611cb499 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a7e6bbc2e9e4e738466019e611cb499" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 34067 } }
I20260926 15:09:13.170797 16077 leader_election.cc:302] T 00000000000000000000000000000000 P 6a7e6bbc2e9e4e738466019e611cb499 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6a7e6bbc2e9e4e738466019e611cb499; no voters:
I20260926 15:09:13.171657 16077 leader_election.cc:288] T 00000000000000000000000000000000 P 6a7e6bbc2e9e4e738466019e611cb499 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 15:09:13.171839 16082 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6a7e6bbc2e9e4e738466019e611cb499 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:09:13.172556 16082 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6a7e6bbc2e9e4e738466019e611cb499 [term 1 LEADER]: Becoming Leader. State: Replica: 6a7e6bbc2e9e4e738466019e611cb499, State: Running, Role: LEADER
I20260926 15:09:13.173130 16082 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6a7e6bbc2e9e4e738466019e611cb499 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a7e6bbc2e9e4e738466019e611cb499" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 34067 } }
I20260926 15:09:13.173761 16077 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6a7e6bbc2e9e4e738466019e611cb499 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:09:13.175781 16083 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a7e6bbc2e9e4e738466019e611cb499 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6a7e6bbc2e9e4e738466019e611cb499" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a7e6bbc2e9e4e738466019e611cb499" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 34067 } } }
I20260926 15:09:13.176124 16083 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a7e6bbc2e9e4e738466019e611cb499 [sys.catalog]: This master's current role is: LEADER
I20260926 15:09:13.177744 16084 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a7e6bbc2e9e4e738466019e611cb499 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6a7e6bbc2e9e4e738466019e611cb499. Latest consensus state: current_term: 1 leader_uuid: "6a7e6bbc2e9e4e738466019e611cb499" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a7e6bbc2e9e4e738466019e611cb499" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 34067 } } }
I20260926 15:09:13.178114 16084 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a7e6bbc2e9e4e738466019e611cb499 [sys.catalog]: This master's current role is: LEADER
I20260926 15:09:13.178747 16089 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:09:13.184690 16089 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 15:09:13.193894 16089 catalog_manager.cc:1347] Generated new cluster ID: 6914444a43434f2180ef12bed3a0350b
I20260926 15:09:13.194025 16089 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 15:09:13.213673 16089 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 15:09:13.215255 16089 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:09:13.223803 16089 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6a7e6bbc2e9e4e738466019e611cb499: Generated new TSK 0
I20260926 15:09:13.224838 16089 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.34.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.34.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.34.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.193.keytab.
Entry for principal kudu/127.12.34.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.193.keytab.
I20260926 15:09:13.272632 12427 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8uoJGv/build/asan/bin/kudu
/tmp/dist-test-task8uoJGv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435318278681-12427-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435318278681-12427-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435318278681-12427-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435318278681-12427-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.34.193:0
--local_ip_for_outbound_sockets=127.12.34.193
--webserver_interface=127.12.34.193
--webserver_port=0
--tserver_master_addrs=127.12.34.254:34067
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.193.keytab
--principal=kudu/127.12.34.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.34.212:45563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435318278681-12427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435318278681-12427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435318278681-12427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:09:13.594642 16103 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:09:13.594954 16103 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:09:13.595120 16103 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:09:13.603626 16103 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:09:13.603796 16103 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:09:13.603986 16103 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.34.193
I20260926 15:09:13.614230 16103 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.34.212:45563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435318278681-12427-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435318278681-12427-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.193.keytab
--principal=kudu/127.12.34.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.34.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435318278681-12427-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.34.193
--webserver_port=0
--tserver_master_addrs=127.12.34.254:34067
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.34.193
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435318278681-12427-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:09:13.615595 16103 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:09:13.616936 16103 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:09:13.629987 16108 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:09:13.630092 16109 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:09:13.630353 16111 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:09:13.631569 16103 server_base.cc:1034] running on GCE node
I20260926 15:09:13.632195 16103 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:09:13.633224 16103 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:09:13.634421 16103 hybrid_clock.cc:648] HybridClock initialized: now 1790435353634388 us; error 61 us; skew 500 ppm
I20260926 15:09:13.634735 16103 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:09:13 dist-test-slave-865v krb5kdc[15997](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435353, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:09:13.638036 16103 init.cc:377] Logged in from keytab as kudu/127.12.34.193@KRBTEST.COM (short username kudu)
I20260926 15:09:13.639612 16103 webserver.cc:460] Webserver started at http://127.12.34.193:35581/ using document root <none> and password file <none>
I20260926 15:09:13.640309 16103 fs_manager.cc:359] Metadata directory not provided
I20260926 15:09:13.640407 16103 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:09:13.640734 16103 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:09:13.643682 16103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435318278681-12427-0/minicluster-data/ts-0/data/instance:
uuid: "73225ccb498843b8a695cd1420eb4457"
format_stamp: "Formatted at 2026-09-26 15:09:13 on dist-test-slave-865v"
server_key: "fad4106efa044a60d1449f3c6f07c4e4"
server_key_iv: "6ca4c67998714098549dcda5f3d2bab7"
server_key_version: "encryptionkey@0"
I20260926 15:09:13.644464 16103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435318278681-12427-0/minicluster-data/ts-0/wal/instance:
uuid: "73225ccb498843b8a695cd1420eb4457"
format_stamp: "Formatted at 2026-09-26 15:09:13 on dist-test-slave-865v"
server_key: "fad4106efa044a60d1449f3c6f07c4e4"
server_key_iv: "6ca4c67998714098549dcda5f3d2bab7"
server_key_version: "encryptionkey@0"
I20260926 15:09:13.649657 16103 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260926 15:09:13.653069 16118 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:09:13.654724 16103 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 15:09:13.654851 16103 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435318278681-12427-0/minicluster-data/ts-0/data,/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435318278681-12427-0/minicluster-data/ts-0/wal
uuid: "73225ccb498843b8a695cd1420eb4457"
format_stamp: "Formatted at 2026-09-26 15:09:13 on dist-test-slave-865v"
server_key: "fad4106efa044a60d1449f3c6f07c4e4"
server_key_iv: "6ca4c67998714098549dcda5f3d2bab7"
server_key_version: "encryptionkey@0"
I20260926 15:09:13.655007 16103 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435318278681-12427-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435318278681-12427-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435318278681-12427-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:09:13.677019 16103 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:09:13.681509 16103 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:09:13.681752 16103 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:09:13.682665 16103 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:09:13.684110 16103 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:09:13.684182 16103 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:09:13.684271 16103 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:09:13.684314 16103 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:09:13.719143 16103 rpc_server.cc:307] RPC server started. Bound to: 127.12.34.193:33035
I20260926 15:09:13.719249 16230 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.34.193:33035 every 8 connection(s)
I20260926 15:09:13.720399 16103 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435318278681-12427-0/minicluster-data/ts-0/data/info.pb
I20260926 15:09:13.723318 12427 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8uoJGv/build/asan/bin/kudu as pid 16103
I20260926 15:09:13.723569 12427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435318278681-12427-0/minicluster-data/ts-0/wal/instance
I20260926 15:09:13.724303 12427 external_mini_cluster.cc:1428] Setting key d0fe3a44d02e604afb6eb516452deece
Sep 26 15:09:13 dist-test-slave-865v krb5kdc[15997](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435353, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.193@KRBTEST.COM for kudu/127.12.34.254@KRBTEST.COM
I20260926 15:09:13.744123 16233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:13.723164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.193:43657 (local address 127.12.34.254:34067)
0926 15:09:13.725695 (+ 2531us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:13.725701 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:13.725723 (+ 22us) server_negotiation.cc:407] Connection header received
0926 15:09:13.727481 (+ 1758us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:13.727485 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:13.727553 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:13.727645 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:09:13.729703 (+ 2058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:13.730380 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:13.732830 (+ 2450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:13.733595 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:13.733747 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:13.737105 (+ 3358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:09:13.737115 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:09:13.737122 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:09:13.737166 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:09:13.740723 (+ 3557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:13.741519 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:13.741528 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:13.741534 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:13.741687 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:13.742111 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:13.742119 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:13.742124 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:13.742505 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:09:13.742664 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0926 15:09:13.743442 (+ 778us) server_negotiation.cc:299] Negotiation successful
0926 15:09:13.743685 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2422,"thread_start_us":2329,"threads_started":1}
WARNING: no policy specified for kudu/127.12.34.194@KRBTEST.COM; defaulting to no policy
I20260926 15:09:13.744836 16234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:13.723883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:34067 (local address 127.12.34.193:43657)
0926 15:09:13.724855 (+ 972us) negotiation.cc:107] Waiting for socket to connect
0926 15:09:13.724919 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0926 15:09:13.727173 (+ 2254us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:09:13.727936 (+ 763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:09:13.727950 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:09:13.728714 (+ 764us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:09:13.729543 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:13.729568 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:13.730521 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:13.730529 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:13.732678 (+ 2149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:13.732690 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:13.733823 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:13.733829 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:13.734038 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:13.734765 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:09:13.734792 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:09:13.736853 (+ 2061us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:09:13.740865 (+ 4012us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:09:13.740874 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:09:13.740890 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:09:13.741397 (+ 507us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:09:13.741806 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:09:13.741810 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:09:13.741812 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:09:13.741985 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:09:13.742695 (+ 710us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:09:13.742705 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:09:13.743049 (+ 344us) client_negotiation.cc:769] Sending connection context
0926 15:09:13.743282 (+ 233us) client_negotiation.cc:240] Negotiation successful
0926 15:09:13.743670 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":659,"thread_start_us":377,"threads_started":1}
I20260926 15:09:13.746323 16231 heartbeater.cc:347] Connected to a master server at 127.12.34.254:34067
I20260926 15:09:13.746657 16231 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.34.194@KRBTEST.COM" created.
I20260926 15:09:13.747525 16231 heartbeater.cc:511] Master 127.12.34.254:34067 requested a full tablet report, sending...
I20260926 15:09:13.749696 16042 ts_manager.cc:194] Registered new tserver with Master: 73225ccb498843b8a695cd1420eb4457 (127.12.34.193:33035)
I20260926 15:09:13.751096 16042 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.34.193@KRBTEST.COM'} at 127.12.34.193:43657
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.34.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.194.keytab.
Entry for principal kudu/127.12.34.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.194.keytab.
I20260926 15:09:13.772704 12427 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8uoJGv/build/asan/bin/kudu
/tmp/dist-test-task8uoJGv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435318278681-12427-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435318278681-12427-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435318278681-12427-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435318278681-12427-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.34.194:0
--local_ip_for_outbound_sockets=127.12.34.194
--webserver_interface=127.12.34.194
--webserver_port=0
--tserver_master_addrs=127.12.34.254:34067
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.194.keytab
--principal=kudu/127.12.34.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.34.212:45563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435318278681-12427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435318278681-12427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435318278681-12427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:09:14.083583 16237 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:09:14.083848 16237 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:09:14.084044 16237 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:09:14.091223 16237 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:09:14.091318 16237 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:09:14.091516 16237 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.34.194
I20260926 15:09:14.100906 16237 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.34.212:45563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435318278681-12427-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435318278681-12427-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.194.keytab
--principal=kudu/127.12.34.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.34.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435318278681-12427-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.34.194
--webserver_port=0
--tserver_master_addrs=127.12.34.254:34067
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.34.194
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435318278681-12427-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:09:14.102165 16237 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:09:14.103408 16237 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:09:14.116258 16243 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:09:14.116285 16242 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:09:14.116547 16245 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:09:14.117373 16237 server_base.cc:1034] running on GCE node
I20260926 15:09:14.117939 16237 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:09:14.118856 16237 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:09:14.120069 16237 hybrid_clock.cc:648] HybridClock initialized: now 1790435354120028 us; error 62 us; skew 500 ppm
I20260926 15:09:14.120350 16237 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:09:14 dist-test-slave-865v krb5kdc[15997](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435354, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:09:14.123476 16237 init.cc:377] Logged in from keytab as kudu/127.12.34.194@KRBTEST.COM (short username kudu)
I20260926 15:09:14.124963 16237 webserver.cc:460] Webserver started at http://127.12.34.194:37897/ using document root <none> and password file <none>
I20260926 15:09:14.125648 16237 fs_manager.cc:359] Metadata directory not provided
I20260926 15:09:14.125782 16237 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:09:14.126113 16237 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:09:14.128899 16237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435318278681-12427-0/minicluster-data/ts-1/data/instance:
uuid: "a3232573b9064f7db82449f97f5b8289"
format_stamp: "Formatted at 2026-09-26 15:09:14 on dist-test-slave-865v"
server_key: "62e9f4f15521a6d919d0557d6eb4593b"
server_key_iv: "c30bd7262af3044272cc0e9ec4728f15"
server_key_version: "encryptionkey@0"
I20260926 15:09:14.129655 16237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435318278681-12427-0/minicluster-data/ts-1/wal/instance:
uuid: "a3232573b9064f7db82449f97f5b8289"
format_stamp: "Formatted at 2026-09-26 15:09:14 on dist-test-slave-865v"
server_key: "62e9f4f15521a6d919d0557d6eb4593b"
server_key_iv: "c30bd7262af3044272cc0e9ec4728f15"
server_key_version: "encryptionkey@0"
I20260926 15:09:14.134570 16237 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260926 15:09:14.138056 16252 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:09:14.139477 16237 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 15:09:14.139643 16237 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435318278681-12427-0/minicluster-data/ts-1/data,/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435318278681-12427-0/minicluster-data/ts-1/wal
uuid: "a3232573b9064f7db82449f97f5b8289"
format_stamp: "Formatted at 2026-09-26 15:09:14 on dist-test-slave-865v"
server_key: "62e9f4f15521a6d919d0557d6eb4593b"
server_key_iv: "c30bd7262af3044272cc0e9ec4728f15"
server_key_version: "encryptionkey@0"
I20260926 15:09:14.139843 16237 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435318278681-12427-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435318278681-12427-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435318278681-12427-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:09:14.154943 16237 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:09:14.158953 16237 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:09:14.159250 16237 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:09:14.160198 16237 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:09:14.161563 16237 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:09:14.161671 16237 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:09:14.161824 16237 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:09:14.161895 16237 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:09:14.193714 16237 rpc_server.cc:307] RPC server started. Bound to: 127.12.34.194:43855
I20260926 15:09:14.193758 16364 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.34.194:43855 every 8 connection(s)
I20260926 15:09:14.194954 16237 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435318278681-12427-0/minicluster-data/ts-1/data/info.pb
I20260926 15:09:14.200206 12427 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8uoJGv/build/asan/bin/kudu as pid 16237
I20260926 15:09:14.200461 12427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435318278681-12427-0/minicluster-data/ts-1/wal/instance
I20260926 15:09:14.200937 12427 external_mini_cluster.cc:1428] Setting key 48c3dedb7f0b8cf333fa7f57449e7311
Sep 26 15:09:14 dist-test-slave-865v krb5kdc[15997](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435354, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.194@KRBTEST.COM for kudu/127.12.34.254@KRBTEST.COM
I20260926 15:09:14.215875 16233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:14.197679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.194:57727 (local address 127.12.34.254:34067)
0926 15:09:14.197894 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:14.197901 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:14.199058 (+ 1157us) server_negotiation.cc:407] Connection header received
0926 15:09:14.201031 (+ 1973us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:14.201037 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:14.201162 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:14.201283 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:09:14.203118 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:14.204004 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:14.205697 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:14.206141 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:14.206285 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:14.209903 (+ 3618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:09:14.209908 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:09:14.209912 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:09:14.209943 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:09:14.212695 (+ 2752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:14.213507 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:14.213515 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:14.213519 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:14.213658 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:14.214097 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:14.214107 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:14.214109 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:14.214359 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:09:14.214517 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0926 15:09:14.215187 (+ 670us) server_negotiation.cc:299] Negotiation successful
0926 15:09:14.215516 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260926 15:09:14.216997 16367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:14.198178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:34067 (local address 127.12.34.194:57727)
0926 15:09:14.198893 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0926 15:09:14.198944 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 15:09:14.200756 (+ 1812us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:09:14.201523 (+ 767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:09:14.201531 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:09:14.202236 (+ 705us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:09:14.202943 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:14.202962 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:14.204163 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:14.204172 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:14.205546 (+ 1374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:14.205558 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:14.206380 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:14.206388 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:14.206623 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:14.207354 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:09:14.207377 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:09:14.209678 (+ 2301us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:09:14.212844 (+ 3166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:09:14.212855 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:09:14.212871 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:09:14.213386 (+ 515us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:09:14.213774 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:09:14.213780 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:09:14.213784 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:09:14.213971 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:09:14.214490 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:09:14.214498 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:09:14.214834 (+ 336us) client_negotiation.cc:769] Sending connection context
0926 15:09:14.215093 (+ 259us) client_negotiation.cc:240] Negotiation successful
0926 15:09:14.215732 (+ 639us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":287,"threads_started":1}
WARNING: no policy specified for kudu/127.12.34.195@KRBTEST.COM; defaulting to no policy
I20260926 15:09:14.218580 16365 heartbeater.cc:347] Connected to a master server at 127.12.34.254:34067
I20260926 15:09:14.218947 16365 heartbeater.cc:464] Registering TS with master...
I20260926 15:09:14.219662 16365 heartbeater.cc:511] Master 127.12.34.254:34067 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.34.195@KRBTEST.COM" created.
I20260926 15:09:14.221223 16042 ts_manager.cc:194] Registered new tserver with Master: a3232573b9064f7db82449f97f5b8289 (127.12.34.194:43855)
I20260926 15:09:14.222074 16042 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.34.194@KRBTEST.COM'} at 127.12.34.194:57727
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.34.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.195.keytab.
Entry for principal kudu/127.12.34.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.195.keytab.
I20260926 15:09:14.244105 12427 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8uoJGv/build/asan/bin/kudu
/tmp/dist-test-task8uoJGv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435318278681-12427-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435318278681-12427-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435318278681-12427-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435318278681-12427-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.34.195:0
--local_ip_for_outbound_sockets=127.12.34.195
--webserver_interface=127.12.34.195
--webserver_port=0
--tserver_master_addrs=127.12.34.254:34067
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.195.keytab
--principal=kudu/127.12.34.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.34.212:45563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435318278681-12427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435318278681-12427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435318278681-12427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:09:14.559839 16370 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:09:14.560081 16370 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:09:14.560235 16370 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:09:14.567296 16370 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:09:14.567443 16370 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:09:14.567627 16370 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.34.195
I20260926 15:09:14.577958 16370 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.34.212:45563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435318278681-12427-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435318278681-12427-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.195.keytab
--principal=kudu/127.12.34.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.34.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435318278681-12427-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.34.195
--webserver_port=0
--tserver_master_addrs=127.12.34.254:34067
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.34.195
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435318278681-12427-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:09:14.579170 16370 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:09:14.580511 16370 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:09:14.593321 16376 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:09:14.593468 16378 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:09:14.593847 16375 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:09:14.594101 16370 server_base.cc:1034] running on GCE node
I20260926 15:09:14.594668 16370 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:09:14.595525 16370 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:09:14.596760 16370 hybrid_clock.cc:648] HybridClock initialized: now 1790435354596706 us; error 58 us; skew 500 ppm
I20260926 15:09:14.597158 16370 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:09:14 dist-test-slave-865v krb5kdc[15997](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435354, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:09:14.600225 16370 init.cc:377] Logged in from keytab as kudu/127.12.34.195@KRBTEST.COM (short username kudu)
I20260926 15:09:14.601747 16370 webserver.cc:460] Webserver started at http://127.12.34.195:46203/ using document root <none> and password file <none>
I20260926 15:09:14.602388 16370 fs_manager.cc:359] Metadata directory not provided
I20260926 15:09:14.602560 16370 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:09:14.602851 16370 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:09:14.605587 16370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435318278681-12427-0/minicluster-data/ts-2/data/instance:
uuid: "b85119f6d53742e68870b4342f055374"
format_stamp: "Formatted at 2026-09-26 15:09:14 on dist-test-slave-865v"
server_key: "2057695f641ba2bd4a769413a60ddd85"
server_key_iv: "9db22ac01c037d0cc199c749906522b6"
server_key_version: "encryptionkey@0"
I20260926 15:09:14.606316 16370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435318278681-12427-0/minicluster-data/ts-2/wal/instance:
uuid: "b85119f6d53742e68870b4342f055374"
format_stamp: "Formatted at 2026-09-26 15:09:14 on dist-test-slave-865v"
server_key: "2057695f641ba2bd4a769413a60ddd85"
server_key_iv: "9db22ac01c037d0cc199c749906522b6"
server_key_version: "encryptionkey@0"
I20260926 15:09:14.611176 16370 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 15:09:14.614437 16385 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:09:14.615626 16370 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 15:09:14.615794 16370 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435318278681-12427-0/minicluster-data/ts-2/data,/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435318278681-12427-0/minicluster-data/ts-2/wal
uuid: "b85119f6d53742e68870b4342f055374"
format_stamp: "Formatted at 2026-09-26 15:09:14 on dist-test-slave-865v"
server_key: "2057695f641ba2bd4a769413a60ddd85"
server_key_iv: "9db22ac01c037d0cc199c749906522b6"
server_key_version: "encryptionkey@0"
I20260926 15:09:14.615999 16370 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435318278681-12427-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435318278681-12427-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435318278681-12427-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:09:14.635562 16370 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:09:14.639801 16370 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:09:14.640110 16370 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:09:14.641047 16370 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:09:14.642302 16370 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:09:14.642403 16370 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:09:14.642547 16370 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:09:14.642616 16370 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:09:14.676357 16370 rpc_server.cc:307] RPC server started. Bound to: 127.12.34.195:35263
I20260926 15:09:14.676397 16497 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.34.195:35263 every 8 connection(s)
I20260926 15:09:14.677619 16370 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435318278681-12427-0/minicluster-data/ts-2/data/info.pb
I20260926 15:09:14.682039 12427 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8uoJGv/build/asan/bin/kudu as pid 16370
I20260926 15:09:14.682261 12427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790435318278681-12427-0/minicluster-data/ts-2/wal/instance
I20260926 15:09:14.682691 12427 external_mini_cluster.cc:1428] Setting key 0a7d43754e318897605cbe398c27f7af
Sep 26 15:09:14 dist-test-slave-865v krb5kdc[15997](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435354, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.195@KRBTEST.COM for kudu/127.12.34.254@KRBTEST.COM
I20260926 15:09:14.698439 16233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:14.680173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.195:41927 (local address 127.12.34.254:34067)
0926 15:09:14.680378 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:14.680382 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:14.681877 (+ 1495us) server_negotiation.cc:407] Connection header received
0926 15:09:14.684210 (+ 2333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:14.684215 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:14.684321 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:14.684439 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:09:14.686289 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:14.687063 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:14.688696 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:14.689194 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:14.689348 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:14.692841 (+ 3493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:09:14.692847 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:09:14.692857 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:09:14.692889 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:09:14.695301 (+ 2412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:14.696019 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:14.696025 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:14.696029 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:14.696144 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:14.696568 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:14.696574 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:14.696578 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:14.696859 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:09:14.697018 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0926 15:09:14.697834 (+ 816us) server_negotiation.cc:299] Negotiation successful
0926 15:09:14.698043 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 15:09:14.699404 16500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:14.680911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:34067 (local address 127.12.34.195:41927)
0926 15:09:14.681696 (+ 785us) negotiation.cc:107] Waiting for socket to connect
0926 15:09:14.681735 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 15:09:14.683947 (+ 2212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:09:14.684722 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:09:14.684736 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:09:14.685543 (+ 807us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:09:14.686112 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:14.686130 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:14.687217 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:14.687230 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:14.688545 (+ 1315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:14.688557 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:14.689442 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:14.689452 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:14.689692 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:14.690368 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:09:14.690395 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:09:14.692621 (+ 2226us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:09:14.695442 (+ 2821us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:09:14.695451 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:09:14.695468 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:09:14.695891 (+ 423us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:09:14.696250 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:09:14.696256 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:09:14.696260 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:09:14.696451 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:09:14.697003 (+ 552us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:09:14.697011 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:09:14.697489 (+ 478us) client_negotiation.cc:769] Sending connection context
0926 15:09:14.697778 (+ 289us) client_negotiation.cc:240] Negotiation successful
0926 15:09:14.698350 (+ 572us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":337,"threads_started":1}
I20260926 15:09:14.700700 16498 heartbeater.cc:347] Connected to a master server at 127.12.34.254:34067
I20260926 15:09:14.701016 16498 heartbeater.cc:464] Registering TS with master...
I20260926 15:09:14.701751 16498 heartbeater.cc:511] Master 127.12.34.254:34067 requested a full tablet report, sending...
I20260926 15:09:14.703090 16042 ts_manager.cc:194] Registered new tserver with Master: b85119f6d53742e68870b4342f055374 (127.12.34.195:35263)
I20260926 15:09:14.703850 16042 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.34.195@KRBTEST.COM'} at 127.12.34.195:41927
I20260926 15:09:14.710120 12427 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 15:09:14 dist-test-slave-865v krb5kdc[15997](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435354, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 15:09:14 dist-test-slave-865v krb5kdc[15997](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435354, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.34.193@KRBTEST.COM
I20260926 15:09:14.754066 16231 heartbeater.cc:503] Master 127.12.34.254:34067 was elected leader, sending a full tablet report...
I20260926 15:09:14.758623 16505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:14.743484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43016 (local address 127.12.34.193:33035)
0926 15:09:14.744163 (+ 679us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:14.744170 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:14.744212 (+ 42us) server_negotiation.cc:407] Connection header received
0926 15:09:14.744278 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:14.744284 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:14.744493 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:14.744620 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:09:14.746007 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:14.746995 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:14.748174 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:14.748725 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:14.748817 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:14.752859 (+ 4042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:09:14.752871 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:09:14.752884 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:09:14.752916 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:09:14.756094 (+ 3178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:14.756718 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:14.756725 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:14.756727 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:14.756834 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:14.757238 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:14.757241 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:14.757243 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:14.757547 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:09:14.757756 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0926 15:09:14.758151 (+ 395us) server_negotiation.cc:299] Negotiation successful
0926 15:09:14.758325 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":499,"thread_start_us":350,"threads_started":1}
Sep 26 15:09:14 dist-test-slave-865v krb5kdc[15997](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435354, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.34.254@KRBTEST.COM
I20260926 15:09:14.777004 16233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:14.763054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44660 (local address 127.12.34.254:34067)
0926 15:09:14.763275 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:14.763280 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:14.763390 (+ 110us) server_negotiation.cc:407] Connection header received
0926 15:09:14.763585 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:14.763590 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:14.763656 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:14.763741 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:09:14.765015 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:14.765802 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:14.766997 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:14.767394 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:14.767487 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:14.771344 (+ 3857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:09:14.771349 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:09:14.771352 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:09:14.771381 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:09:14.774088 (+ 2707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:14.774969 (+ 881us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:14.774973 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:14.774975 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:14.775112 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:14.775480 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:14.775484 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:14.775486 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:14.775804 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:09:14.776091 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0926 15:09:14.776508 (+ 417us) server_negotiation.cc:299] Negotiation successful
0926 15:09:14.776761 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
W20260926 15:09:14.777805 16042 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44660
I20260926 15:09:14.779063 12427 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 16103
I20260926 15:09:14.792891 16505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:14.780122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43024 (local address 127.12.34.193:33035)
0926 15:09:14.780306 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:14.780312 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:14.780527 (+ 215us) server_negotiation.cc:407] Connection header received
0926 15:09:14.780613 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:14.780617 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:14.780710 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:14.780801 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:09:14.782184 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:14.783077 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:14.784531 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:14.785355 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:14.785651 (+ 296us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:14.787831 (+ 2180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:09:14.787837 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:09:14.787840 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:09:14.787871 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:09:14.790447 (+ 2576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:14.791099 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:14.791104 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:14.791107 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:14.791227 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:14.791592 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:14.791598 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:14.791601 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:14.791839 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:09:14.792091 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0926 15:09:14.792371 (+ 280us) server_negotiation.cc:299] Negotiation successful
0926 15:09:14.792583 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 15:09:14.793596 16226 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:09:14.876430 12427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8uoJGv/build/asan/bin/kudu with pid 16103
I20260926 15:09:14.892162 12427 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 16237
Sep 26 15:09:14 dist-test-slave-865v krb5kdc[15997](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435354, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.34.194@KRBTEST.COM
I20260926 15:09:14.908759 16510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:14.893202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44792 (local address 127.12.34.194:43855)
0926 15:09:14.893732 (+ 530us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:14.893740 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:14.893765 (+ 25us) server_negotiation.cc:407] Connection header received
0926 15:09:14.893823 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:14.893829 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:14.894056 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:14.894184 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:09:14.895267 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:14.896254 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:14.897942 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:14.898731 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:14.898893 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:14.903213 (+ 4320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:09:14.903226 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:09:14.903241 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:09:14.903295 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:09:14.906195 (+ 2900us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:14.906841 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:14.906848 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:14.906850 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:14.907004 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:14.907320 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:14.907324 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:14.907325 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:14.907656 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:09:14.907872 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0926 15:09:14.908182 (+ 310us) server_negotiation.cc:299] Negotiation successful
0926 15:09:14.908367 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":401,"thread_start_us":266,"threads_started":1}
I20260926 15:09:14.910099 16360 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:09:14.990931 12427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8uoJGv/build/asan/bin/kudu with pid 16237
I20260926 15:09:15.006642 12427 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 16370
Sep 26 15:09:15 dist-test-slave-865v krb5kdc[15997](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435354, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.34.195@KRBTEST.COM
I20260926 15:09:15.023320 16512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:15.007889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50046 (local address 127.12.34.195:35263)
0926 15:09:15.008580 (+ 691us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:15.008587 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:15.008617 (+ 30us) server_negotiation.cc:407] Connection header received
0926 15:09:15.008678 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:15.008684 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:15.008913 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:15.009044 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:09:15.010243 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:15.011300 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:15.012563 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:15.013205 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:15.013302 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:15.017598 (+ 4296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:09:15.017609 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:09:15.017624 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:09:15.017663 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:09:15.020572 (+ 2909us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:15.021359 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:15.021366 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:15.021369 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:15.021517 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:15.021888 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:15.021891 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:15.021893 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:15.022231 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:09:15.022466 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0926 15:09:15.022745 (+ 279us) server_negotiation.cc:299] Negotiation successful
0926 15:09:15.022982 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":549,"thread_start_us":377,"threads_started":1}
I20260926 15:09:15.024523 16493 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:09:15.108995 12427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8uoJGv/build/asan/bin/kudu with pid 16370
I20260926 15:09:15.124572 12427 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 16010
I20260926 15:09:15.125847 16072 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:09:15.227151 12427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8uoJGv/build/asan/bin/kudu with pid 16010
2026-09-26T15:09:15Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3824 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435318278681-12427-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 15:09:15 dist-test-slave-865v krb5kdc[16517](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 15:09:15 dist-test-slave-865v krb5kdc[16517](info): set up 2 sockets
Sep 26 15:09:15 dist-test-slave-865v krb5kdc[16517](info): commencing operation
krb5kdc: starting...
W20260926 15:09:16.329563 12427 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.006s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435318278681-12427-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435318278681-12427-0/krb5kdc/test-user.keytab.
Sep 26 15:09:16 dist-test-slave-865v krb5kdc[16517](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435356, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T15:09:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T15:09:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.34.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.34.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.34.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.254.keytab.
Entry for principal kudu/127.12.34.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.254.keytab.
I20260926 15:09:16.528146 12427 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8uoJGv/build/asan/bin/kudu
/tmp/dist-test-task8uoJGv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435318278681-12427-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435318278681-12427-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435318278681-12427-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.34.254:45317
--webserver_interface=127.12.34.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.254.keytab
--principal=kudu/127.12.34.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.34.212:42669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.34.254:45317
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435318278681-12427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435318278681-12427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435318278681-12427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:09:16.842726 16530 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:09:16.843070 16530 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:09:16.843241 16530 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:09:16.850497 16530 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:09:16.850638 16530 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:09:16.850744 16530 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:09:16.850845 16530 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:09:16.850944 16530 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:09:16.860754 16530 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.34.212:42669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435318278681-12427-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435318278681-12427-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.34.254:45317
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.254.keytab
--principal=kudu/127.12.34.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.34.254:45317
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.34.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435318278681-12427-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:09:16.862078 16530 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:09:16.863390 16530 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:09:16.875396 16536 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:09:16.875468 16535 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:09:16.875499 16538 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:09:16.876724 16530 server_base.cc:1034] running on GCE node
I20260926 15:09:16.877487 16530 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:09:16.878789 16530 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:09:16.879994 16530 hybrid_clock.cc:648] HybridClock initialized: now 1790435356879958 us; error 34 us; skew 500 ppm
I20260926 15:09:16.880285 16530 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:09:16 dist-test-slave-865v krb5kdc[16517](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435356, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:09:16.883330 16530 init.cc:377] Logged in from keytab as kudu/127.12.34.254@KRBTEST.COM (short username kudu)
I20260926 15:09:16.884459 16530 fs_manager.cc:359] Metadata directory not provided
I20260926 15:09:16.884542 16530 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:09:16.884871 16530 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:09:16.887832 16530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435318278681-12427-0/minicluster-data/master-0/data/instance:
uuid: "affb42e14eb5434b90bc13b1815a3b90"
format_stamp: "Formatted at 2026-09-26 15:09:16 on dist-test-slave-865v"
server_key: "6dc6ba606148f2a4ee46fd3992ed8fcb"
server_key_iv: "0f64a2905c6bc16e3acab06c9d2d1f7c"
server_key_version: "encryptionkey@0"
I20260926 15:09:16.888526 16530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435318278681-12427-0/minicluster-data/master-0/wal/instance:
uuid: "affb42e14eb5434b90bc13b1815a3b90"
format_stamp: "Formatted at 2026-09-26 15:09:16 on dist-test-slave-865v"
server_key: "6dc6ba606148f2a4ee46fd3992ed8fcb"
server_key_iv: "0f64a2905c6bc16e3acab06c9d2d1f7c"
server_key_version: "encryptionkey@0"
I20260926 15:09:16.893148 16530 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.003s
I20260926 15:09:16.896405 16544 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:09:16.897802 16530 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 15:09:16.897985 16530 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435318278681-12427-0/minicluster-data/master-0/data,/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435318278681-12427-0/minicluster-data/master-0/wal
uuid: "affb42e14eb5434b90bc13b1815a3b90"
format_stamp: "Formatted at 2026-09-26 15:09:16 on dist-test-slave-865v"
server_key: "6dc6ba606148f2a4ee46fd3992ed8fcb"
server_key_iv: "0f64a2905c6bc16e3acab06c9d2d1f7c"
server_key_version: "encryptionkey@0"
I20260926 15:09:16.898211 16530 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435318278681-12427-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435318278681-12427-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435318278681-12427-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:09:16.916800 16530 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:09:16.921000 16530 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:09:16.921311 16530 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:09:16.940699 16530 rpc_server.cc:307] RPC server started. Bound to: 127.12.34.254:45317
I20260926 15:09:16.940719 16595 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.34.254:45317 every 8 connection(s)
I20260926 15:09:16.941658 16530 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
I20260926 15:09:16.945307 16596 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:09:16.946256 12427 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8uoJGv/build/asan/bin/kudu as pid 16530
I20260926 15:09:16.946525 12427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435318278681-12427-0/minicluster-data/master-0/wal/instance
I20260926 15:09:16.947167 12427 external_mini_cluster.cc:1428] Setting key 47ec904a4b62d88ec46cd713b8c7a5e1
I20260926 15:09:16.959503 16596 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P affb42e14eb5434b90bc13b1815a3b90: Bootstrap starting.
Sep 26 15:09:16 dist-test-slave-865v krb5kdc[16517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435356, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.34.254@KRBTEST.COM
I20260926 15:09:16.963195 16596 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P affb42e14eb5434b90bc13b1815a3b90: Neither blocks nor log segments found. Creating new log.
I20260926 15:09:16.964268 16596 log.cc:824] T 00000000000000000000000000000000 P affb42e14eb5434b90bc13b1815a3b90: Log is configured to *not* fsync() on all Append() calls
I20260926 15:09:16.967123 16596 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P affb42e14eb5434b90bc13b1815a3b90: No bootstrap required, opened a new log
I20260926 15:09:16.969689 16599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:16.949258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59826 (local address 127.12.34.254:45317)
0926 15:09:16.950082 (+ 824us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:16.950093 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:16.950133 (+ 40us) server_negotiation.cc:407] Connection header received
0926 15:09:16.951652 (+ 1519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:16.951672 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:16.952056 (+ 384us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:16.952357 (+ 301us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:09:16.954165 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:16.955423 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:16.957200 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:16.957891 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:16.958014 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:16.962572 (+ 4558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:09:16.962579 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:09:16.962594 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:09:16.962628 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:09:16.965704 (+ 3076us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:16.966382 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:16.966388 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:16.966394 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:16.966544 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:16.966927 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:16.966934 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:16.966942 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:16.967514 (+ 572us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:09:16.967838 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0926 15:09:16.968282 (+ 444us) server_negotiation.cc:299] Negotiation successful
0926 15:09:16.968672 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":623,"thread_start_us":354,"threads_started":1}
I20260926 15:09:16.975196 16596 raft_consensus.cc:348] T 00000000000000000000000000000000 P affb42e14eb5434b90bc13b1815a3b90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "affb42e14eb5434b90bc13b1815a3b90" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 45317 } }
I20260926 15:09:16.975538 16596 raft_consensus.cc:374] T 00000000000000000000000000000000 P affb42e14eb5434b90bc13b1815a3b90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:09:16.975643 16596 raft_consensus.cc:729] T 00000000000000000000000000000000 P affb42e14eb5434b90bc13b1815a3b90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: affb42e14eb5434b90bc13b1815a3b90, State: Initialized, Role: FOLLOWER
I20260926 15:09:16.976158 16596 consensus_queue.cc:260] T 00000000000000000000000000000000 P affb42e14eb5434b90bc13b1815a3b90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "affb42e14eb5434b90bc13b1815a3b90" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 45317 } }
I20260926 15:09:16.976392 16596 raft_consensus.cc:388] T 00000000000000000000000000000000 P affb42e14eb5434b90bc13b1815a3b90 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 15:09:16.976531 16596 raft_consensus.cc:482] T 00000000000000000000000000000000 P affb42e14eb5434b90bc13b1815a3b90 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 15:09:16.976694 16596 raft_consensus.cc:3037] T 00000000000000000000000000000000 P affb42e14eb5434b90bc13b1815a3b90 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:09:16.979005 16596 raft_consensus.cc:504] T 00000000000000000000000000000000 P affb42e14eb5434b90bc13b1815a3b90 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "affb42e14eb5434b90bc13b1815a3b90" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 45317 } }
I20260926 15:09:16.979585 16596 leader_election.cc:302] T 00000000000000000000000000000000 P affb42e14eb5434b90bc13b1815a3b90 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: affb42e14eb5434b90bc13b1815a3b90; no voters:
I20260926 15:09:16.980192 16596 leader_election.cc:288] T 00000000000000000000000000000000 P affb42e14eb5434b90bc13b1815a3b90 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 15:09:16.980337 16601 raft_consensus.cc:2781] T 00000000000000000000000000000000 P affb42e14eb5434b90bc13b1815a3b90 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:09:16.980984 16601 raft_consensus.cc:686] T 00000000000000000000000000000000 P affb42e14eb5434b90bc13b1815a3b90 [term 1 LEADER]: Becoming Leader. State: Replica: affb42e14eb5434b90bc13b1815a3b90, State: Running, Role: LEADER
I20260926 15:09:16.981493 16601 consensus_queue.cc:237] T 00000000000000000000000000000000 P affb42e14eb5434b90bc13b1815a3b90 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "affb42e14eb5434b90bc13b1815a3b90" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 45317 } }
I20260926 15:09:16.982228 16596 sys_catalog.cc:564] T 00000000000000000000000000000000 P affb42e14eb5434b90bc13b1815a3b90 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:09:16.983969 16603 sys_catalog.cc:455] T 00000000000000000000000000000000 P affb42e14eb5434b90bc13b1815a3b90 [sys.catalog]: SysCatalogTable state changed. Reason: New leader affb42e14eb5434b90bc13b1815a3b90. Latest consensus state: current_term: 1 leader_uuid: "affb42e14eb5434b90bc13b1815a3b90" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "affb42e14eb5434b90bc13b1815a3b90" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 45317 } } }
I20260926 15:09:16.984158 16602 sys_catalog.cc:455] T 00000000000000000000000000000000 P affb42e14eb5434b90bc13b1815a3b90 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "affb42e14eb5434b90bc13b1815a3b90" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "affb42e14eb5434b90bc13b1815a3b90" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 45317 } } }
I20260926 15:09:16.985431 16603 sys_catalog.cc:458] T 00000000000000000000000000000000 P affb42e14eb5434b90bc13b1815a3b90 [sys.catalog]: This master's current role is: LEADER
I20260926 15:09:16.985536 16602 sys_catalog.cc:458] T 00000000000000000000000000000000 P affb42e14eb5434b90bc13b1815a3b90 [sys.catalog]: This master's current role is: LEADER
I20260926 15:09:16.987123 16608 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:09:16.992107 16608 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 15:09:17.001109 16608 catalog_manager.cc:1347] Generated new cluster ID: c344adcb37c848c195b6871e93433607
I20260926 15:09:17.001236 16608 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 15:09:17.029358 16608 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 15:09:17.030645 16608 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:09:17.038726 16608 catalog_manager.cc:5868] T 00000000000000000000000000000000 P affb42e14eb5434b90bc13b1815a3b90: Generated new TSK 0
I20260926 15:09:17.039665 16608 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.34.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.34.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.34.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.193.keytab.
Entry for principal kudu/127.12.34.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.193.keytab.
I20260926 15:09:17.097862 12427 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8uoJGv/build/asan/bin/kudu
/tmp/dist-test-task8uoJGv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435318278681-12427-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435318278681-12427-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435318278681-12427-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435318278681-12427-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.34.193:0
--local_ip_for_outbound_sockets=127.12.34.193
--webserver_interface=127.12.34.193
--webserver_port=0
--tserver_master_addrs=127.12.34.254:45317
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.193.keytab
--principal=kudu/127.12.34.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.34.212:42669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435318278681-12427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435318278681-12427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435318278681-12427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:09:17.415869 16622 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:09:17.416122 16622 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:09:17.416278 16622 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:09:17.423352 16622 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:09:17.423454 16622 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:09:17.423648 16622 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.34.193
I20260926 15:09:17.433005 16622 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.34.212:42669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435318278681-12427-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435318278681-12427-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.193.keytab
--principal=kudu/127.12.34.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.34.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435318278681-12427-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.34.193
--webserver_port=0
--tserver_master_addrs=127.12.34.254:45317
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.34.193
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435318278681-12427-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:09:17.434396 16622 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:09:17.435609 16622 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:09:17.447885 16627 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:09:17.447930 16628 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:09:17.449139 16630 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:09:17.449187 16622 server_base.cc:1034] running on GCE node
I20260926 15:09:17.449769 16622 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:09:17.450640 16622 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:09:17.451858 16622 hybrid_clock.cc:648] HybridClock initialized: now 1790435357451817 us; error 36 us; skew 500 ppm
I20260926 15:09:17.452147 16622 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:09:17 dist-test-slave-865v krb5kdc[16517](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435357, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:09:17.455108 16622 init.cc:377] Logged in from keytab as kudu/127.12.34.193@KRBTEST.COM (short username kudu)
I20260926 15:09:17.456197 16622 fs_manager.cc:359] Metadata directory not provided
I20260926 15:09:17.456319 16622 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:09:17.456634 16622 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:09:17.459350 16622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435318278681-12427-0/minicluster-data/ts-0/data/instance:
uuid: "8a094011d4924b2b86082363f5750607"
format_stamp: "Formatted at 2026-09-26 15:09:17 on dist-test-slave-865v"
server_key: "f37bcd1f6a11c8b7ccdc17ce4fb6c648"
server_key_iv: "75e493ba5d6a7ef765931395165ee6f1"
server_key_version: "encryptionkey@0"
I20260926 15:09:17.460111 16622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435318278681-12427-0/minicluster-data/ts-0/wal/instance:
uuid: "8a094011d4924b2b86082363f5750607"
format_stamp: "Formatted at 2026-09-26 15:09:17 on dist-test-slave-865v"
server_key: "f37bcd1f6a11c8b7ccdc17ce4fb6c648"
server_key_iv: "75e493ba5d6a7ef765931395165ee6f1"
server_key_version: "encryptionkey@0"
I20260926 15:09:17.464720 16622 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260926 15:09:17.468094 16636 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:09:17.469345 16622 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 15:09:17.469514 16622 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435318278681-12427-0/minicluster-data/ts-0/data,/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435318278681-12427-0/minicluster-data/ts-0/wal
uuid: "8a094011d4924b2b86082363f5750607"
format_stamp: "Formatted at 2026-09-26 15:09:17 on dist-test-slave-865v"
server_key: "f37bcd1f6a11c8b7ccdc17ce4fb6c648"
server_key_iv: "75e493ba5d6a7ef765931395165ee6f1"
server_key_version: "encryptionkey@0"
I20260926 15:09:17.469714 16622 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435318278681-12427-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435318278681-12427-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435318278681-12427-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:09:17.491283 16622 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:09:17.495325 16622 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:09:17.495605 16622 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:09:17.496515 16622 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:09:17.497872 16622 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:09:17.497958 16622 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:09:17.498083 16622 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:09:17.498170 16622 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:09:17.528923 16622 rpc_server.cc:307] RPC server started. Bound to: 127.12.34.193:35561
I20260926 15:09:17.528973 16748 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.34.193:35561 every 8 connection(s)
I20260926 15:09:17.529760 16622 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435318278681-12427-0/minicluster-data/ts-0/data/info.pb
I20260926 15:09:17.537169 12427 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8uoJGv/build/asan/bin/kudu as pid 16622
I20260926 15:09:17.537411 12427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435318278681-12427-0/minicluster-data/ts-0/wal/instance
I20260926 15:09:17.538184 12427 external_mini_cluster.cc:1428] Setting key d951e735403be29de6f63de4659cec62
Sep 26 15:09:17 dist-test-slave-865v krb5kdc[16517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435357, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.193@KRBTEST.COM for kudu/127.12.34.254@KRBTEST.COM
I20260926 15:09:17.550261 16751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:17.532061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.193:35891 (local address 127.12.34.254:45317)
0926 15:09:17.532471 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:17.532477 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:17.533624 (+ 1147us) server_negotiation.cc:407] Connection header received
0926 15:09:17.535321 (+ 1697us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:17.535325 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:17.535397 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:17.535509 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:09:17.537194 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:17.538065 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:17.539799 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:17.540316 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:17.540411 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:17.544379 (+ 3968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:09:17.544384 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:09:17.544388 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:09:17.544415 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:09:17.547323 (+ 2908us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:17.548013 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:17.548023 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:17.548025 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:17.548133 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:17.548573 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:17.548578 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:17.548581 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:17.548839 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:09:17.549001 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0926 15:09:17.549704 (+ 703us) server_negotiation.cc:299] Negotiation successful
0926 15:09:17.549905 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":187,"threads_started":1}
I20260926 15:09:17.551131 16752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:17.532607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:45317 (local address 127.12.34.193:35891)
0926 15:09:17.533383 (+ 776us) negotiation.cc:107] Waiting for socket to connect
0926 15:09:17.533477 (+ 94us) client_negotiation.cc:174] Beginning negotiation
0926 15:09:17.535083 (+ 1606us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:09:17.535743 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:09:17.535753 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:09:17.536255 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:09:17.536945 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:17.536971 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:17.538210 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:17.538217 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:17.539661 (+ 1444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:17.539674 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:17.540531 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:17.540537 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:17.540823 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:17.541552 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:09:17.541586 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:09:17.544166 (+ 2580us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:09:17.547471 (+ 3305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:09:17.547479 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:09:17.547495 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:09:17.547878 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:09:17.548251 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:09:17.548258 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:09:17.548261 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:09:17.548421 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:09:17.548964 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:09:17.548971 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:09:17.549351 (+ 380us) client_negotiation.cc:769] Sending connection context
0926 15:09:17.549604 (+ 253us) client_negotiation.cc:240] Negotiation successful
0926 15:09:17.550079 (+ 475us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":300,"threads_started":1}
I20260926 15:09:17.552649 16749 heartbeater.cc:347] Connected to a master server at 127.12.34.254:45317
I20260926 15:09:17.552930 16749 heartbeater.cc:464] Registering TS with master...
I20260926 15:09:17.553566 16749 heartbeater.cc:511] Master 127.12.34.254:45317 requested a full tablet report, sending...
I20260926 15:09:17.555751 16561 ts_manager.cc:194] Registered new tserver with Master: 8a094011d4924b2b86082363f5750607 (127.12.34.193:35561)
WARNING: no policy specified for kudu/127.12.34.194@KRBTEST.COM; defaulting to no policy
I20260926 15:09:17.557747 16561 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.34.193@KRBTEST.COM'} at 127.12.34.193:35891
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.34.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.34.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.194.keytab.
Entry for principal kudu/127.12.34.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.194.keytab.
I20260926 15:09:17.584398 12427 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8uoJGv/build/asan/bin/kudu
/tmp/dist-test-task8uoJGv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435318278681-12427-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435318278681-12427-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435318278681-12427-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435318278681-12427-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.34.194:0
--local_ip_for_outbound_sockets=127.12.34.194
--webserver_interface=127.12.34.194
--webserver_port=0
--tserver_master_addrs=127.12.34.254:45317
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.194.keytab
--principal=kudu/127.12.34.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.34.212:42669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435318278681-12427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435318278681-12427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435318278681-12427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:09:17.910811 16755 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:09:17.911083 16755 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:09:17.911183 16755 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:09:17.918672 16755 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:09:17.918776 16755 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:09:17.918890 16755 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.34.194
I20260926 15:09:17.928417 16755 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.34.212:42669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435318278681-12427-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435318278681-12427-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.194.keytab
--principal=kudu/127.12.34.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.34.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435318278681-12427-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.34.194
--webserver_port=0
--tserver_master_addrs=127.12.34.254:45317
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.34.194
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435318278681-12427-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:09:17.929637 16755 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:09:17.930868 16755 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:09:17.943341 16760 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:09:17.943365 16761 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:09:17.943421 16763 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:09:17.946452 16755 server_base.cc:1034] running on GCE node
I20260926 15:09:17.947034 16755 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:09:17.948019 16755 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:09:17.949188 16755 hybrid_clock.cc:648] HybridClock initialized: now 1790435357949137 us; error 48 us; skew 500 ppm
I20260926 15:09:17.949496 16755 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:09:17 dist-test-slave-865v krb5kdc[16517](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435357, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:09:17.952591 16755 init.cc:377] Logged in from keytab as kudu/127.12.34.194@KRBTEST.COM (short username kudu)
I20260926 15:09:17.953786 16755 fs_manager.cc:359] Metadata directory not provided
I20260926 15:09:17.953913 16755 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:09:17.954221 16755 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:09:17.956892 16755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435318278681-12427-0/minicluster-data/ts-1/data/instance:
uuid: "d1ee0420925e4a4a831dd047c84c0b1e"
format_stamp: "Formatted at 2026-09-26 15:09:17 on dist-test-slave-865v"
server_key: "0a383552b9f1911419ed6570114fa23b"
server_key_iv: "8bf9052288bea7dfe11d385044493566"
server_key_version: "encryptionkey@0"
I20260926 15:09:17.957715 16755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435318278681-12427-0/minicluster-data/ts-1/wal/instance:
uuid: "d1ee0420925e4a4a831dd047c84c0b1e"
format_stamp: "Formatted at 2026-09-26 15:09:17 on dist-test-slave-865v"
server_key: "0a383552b9f1911419ed6570114fa23b"
server_key_iv: "8bf9052288bea7dfe11d385044493566"
server_key_version: "encryptionkey@0"
I20260926 15:09:17.962492 16755 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260926 15:09:17.965700 16769 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:09:17.967068 16755 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 15:09:17.967278 16755 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435318278681-12427-0/minicluster-data/ts-1/data,/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435318278681-12427-0/minicluster-data/ts-1/wal
uuid: "d1ee0420925e4a4a831dd047c84c0b1e"
format_stamp: "Formatted at 2026-09-26 15:09:17 on dist-test-slave-865v"
server_key: "0a383552b9f1911419ed6570114fa23b"
server_key_iv: "8bf9052288bea7dfe11d385044493566"
server_key_version: "encryptionkey@0"
I20260926 15:09:17.967577 16755 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435318278681-12427-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435318278681-12427-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435318278681-12427-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:09:17.985488 16755 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:09:17.989789 16755 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:09:17.990038 16755 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:09:17.991017 16755 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:09:17.992272 16755 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:09:17.992368 16755 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:09:17.992451 16755 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:09:17.992498 16755 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:09:18.028993 16755 rpc_server.cc:307] RPC server started. Bound to: 127.12.34.194:42063
I20260926 15:09:18.029083 16881 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.34.194:42063 every 8 connection(s)
I20260926 15:09:18.029871 16755 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435318278681-12427-0/minicluster-data/ts-1/data/info.pb
I20260926 15:09:18.033042 12427 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8uoJGv/build/asan/bin/kudu as pid 16755
I20260926 15:09:18.033268 12427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435318278681-12427-0/minicluster-data/ts-1/wal/instance
I20260926 15:09:18.033735 12427 external_mini_cluster.cc:1428] Setting key 20121f7893dbbb3e33c74f5a3b658811
Sep 26 15:09:18 dist-test-slave-865v krb5kdc[16517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435357, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.194@KRBTEST.COM for kudu/127.12.34.254@KRBTEST.COM
I20260926 15:09:18.053022 16751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:18.032498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.194:40217 (local address 127.12.34.254:45317)
0926 15:09:18.032712 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:18.032720 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:18.034083 (+ 1363us) server_negotiation.cc:407] Connection header received
0926 15:09:18.036323 (+ 2240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:18.036330 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:18.036435 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:18.036557 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:09:18.038101 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:18.039260 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:18.041421 (+ 2161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:18.042125 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:18.042292 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:18.046181 (+ 3889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:09:18.046190 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:09:18.046197 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:09:18.046247 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:09:18.049982 (+ 3735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:18.050745 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:18.050755 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:18.050759 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:18.050894 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:18.051327 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:18.051332 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:18.051341 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:18.051594 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:09:18.051765 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0926 15:09:18.052446 (+ 681us) server_negotiation.cc:299] Negotiation successful
0926 15:09:18.052661 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 15:09:18.054009 16884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:18.032960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:45317 (local address 127.12.34.194:40217)
0926 15:09:18.033903 (+ 943us) negotiation.cc:107] Waiting for socket to connect
0926 15:09:18.033958 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0926 15:09:18.036056 (+ 2098us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:09:18.036736 (+ 680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:09:18.036744 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:09:18.037357 (+ 613us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:09:18.037927 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:18.037949 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:18.039544 (+ 1595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:18.039551 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:18.041268 (+ 1717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:18.041285 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:18.042390 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:18.042395 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:18.042600 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:18.043312 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:09:18.043333 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:09:18.045936 (+ 2603us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:09:18.050129 (+ 4193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:09:18.050137 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:09:18.050150 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:09:18.050595 (+ 445us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:09:18.051018 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:09:18.051024 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:09:18.051027 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:09:18.051202 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:09:18.051732 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:09:18.051740 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:09:18.052089 (+ 349us) client_negotiation.cc:769] Sending connection context
0926 15:09:18.052349 (+ 260us) client_negotiation.cc:240] Negotiation successful
0926 15:09:18.052900 (+ 551us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":656,"thread_start_us":473,"threads_started":1}
WARNING: no policy specified for kudu/127.12.34.195@KRBTEST.COM; defaulting to no policy
I20260926 15:09:18.055565 16882 heartbeater.cc:347] Connected to a master server at 127.12.34.254:45317
I20260926 15:09:18.055886 16882 heartbeater.cc:464] Registering TS with master...
I20260926 15:09:18.056834 16882 heartbeater.cc:511] Master 127.12.34.254:45317 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.34.195@KRBTEST.COM" created.
I20260926 15:09:18.058437 16561 ts_manager.cc:194] Registered new tserver with Master: d1ee0420925e4a4a831dd047c84c0b1e (127.12.34.194:42063)
I20260926 15:09:18.059244 16561 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.34.194@KRBTEST.COM'} at 127.12.34.194:40217
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.34.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.195.keytab.
Entry for principal kudu/127.12.34.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.195.keytab.
I20260926 15:09:18.090327 12427 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8uoJGv/build/asan/bin/kudu
/tmp/dist-test-task8uoJGv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435318278681-12427-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435318278681-12427-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435318278681-12427-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435318278681-12427-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.34.195:0
--local_ip_for_outbound_sockets=127.12.34.195
--webserver_interface=127.12.34.195
--webserver_port=0
--tserver_master_addrs=127.12.34.254:45317
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.195.keytab
--principal=kudu/127.12.34.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.34.212:42669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435318278681-12427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435318278681-12427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435318278681-12427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:09:18.411687 16887 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:09:18.411957 16887 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:09:18.412066 16887 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:09:18.419067 16887 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:09:18.419159 16887 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:09:18.419269 16887 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.34.195
I20260926 15:09:18.428474 16887 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.34.212:42669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435318278681-12427-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435318278681-12427-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.195.keytab
--principal=kudu/127.12.34.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.34.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435318278681-12427-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.34.195
--webserver_port=0
--tserver_master_addrs=127.12.34.254:45317
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.34.195
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435318278681-12427-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:09:18.429790 16887 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:09:18.430956 16887 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:09:18.443598 16892 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:09:18.443567 16893 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:09:18.443531 16895 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:09:18.444499 16887 server_base.cc:1034] running on GCE node
I20260926 15:09:18.445053 16887 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:09:18.446005 16887 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:09:18.447235 16887 hybrid_clock.cc:648] HybridClock initialized: now 1790435358447198 us; error 67 us; skew 500 ppm
I20260926 15:09:18.447569 16887 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:09:18 dist-test-slave-865v krb5kdc[16517](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435358, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:09:18.459703 16887 init.cc:377] Logged in from keytab as kudu/127.12.34.195@KRBTEST.COM (short username kudu)
I20260926 15:09:18.460767 16887 fs_manager.cc:359] Metadata directory not provided
I20260926 15:09:18.460850 16887 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:09:18.461184 16887 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:09:18.463927 16887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435318278681-12427-0/minicluster-data/ts-2/data/instance:
uuid: "245c8d956046409b8cc89fbaf0e9c243"
format_stamp: "Formatted at 2026-09-26 15:09:18 on dist-test-slave-865v"
server_key: "85d5a154a987259f762dd9502ab7f1be"
server_key_iv: "d964ef7eb7a6fa73cc7e7fd781c70dc6"
server_key_version: "encryptionkey@0"
I20260926 15:09:18.464648 16887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435318278681-12427-0/minicluster-data/ts-2/wal/instance:
uuid: "245c8d956046409b8cc89fbaf0e9c243"
format_stamp: "Formatted at 2026-09-26 15:09:18 on dist-test-slave-865v"
server_key: "85d5a154a987259f762dd9502ab7f1be"
server_key_iv: "d964ef7eb7a6fa73cc7e7fd781c70dc6"
server_key_version: "encryptionkey@0"
I20260926 15:09:18.469645 16887 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260926 15:09:18.473217 16901 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:09:18.474706 16887 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 15:09:18.474879 16887 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435318278681-12427-0/minicluster-data/ts-2/data,/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435318278681-12427-0/minicluster-data/ts-2/wal
uuid: "245c8d956046409b8cc89fbaf0e9c243"
format_stamp: "Formatted at 2026-09-26 15:09:18 on dist-test-slave-865v"
server_key: "85d5a154a987259f762dd9502ab7f1be"
server_key_iv: "d964ef7eb7a6fa73cc7e7fd781c70dc6"
server_key_version: "encryptionkey@0"
I20260926 15:09:18.475100 16887 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435318278681-12427-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435318278681-12427-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435318278681-12427-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:09:18.487797 16887 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:09:18.491899 16887 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:09:18.492159 16887 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:09:18.493072 16887 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:09:18.494364 16887 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:09:18.494446 16887 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:09:18.494520 16887 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:09:18.494575 16887 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:09:18.526444 16887 rpc_server.cc:307] RPC server started. Bound to: 127.12.34.195:37409
I20260926 15:09:18.526532 17013 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.34.195:37409 every 8 connection(s)
I20260926 15:09:18.527344 16887 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435318278681-12427-0/minicluster-data/ts-2/data/info.pb
I20260926 15:09:18.529495 12427 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8uoJGv/build/asan/bin/kudu as pid 16887
I20260926 15:09:18.529688 12427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435318278681-12427-0/minicluster-data/ts-2/wal/instance
I20260926 15:09:18.530552 12427 external_mini_cluster.cc:1428] Setting key afff8b7e83ad0fb55c07f37a009ddb94
Sep 26 15:09:18 dist-test-slave-865v krb5kdc[16517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435358, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.195@KRBTEST.COM for kudu/127.12.34.254@KRBTEST.COM
I20260926 15:09:18.547667 16751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:18.531973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.195:36713 (local address 127.12.34.254:45317)
0926 15:09:18.532169 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:18.532176 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:18.532197 (+ 21us) server_negotiation.cc:407] Connection header received
0926 15:09:18.533932 (+ 1735us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:18.533938 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:18.534022 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:18.534141 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:09:18.535772 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:18.536643 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:18.538140 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:18.538622 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:18.538722 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:18.542151 (+ 3429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:09:18.542162 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:09:18.542166 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:09:18.542198 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:09:18.544742 (+ 2544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:18.545634 (+ 892us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:18.545638 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:18.545640 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:18.545822 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:18.546282 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:18.546285 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:18.546287 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:18.546604 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:09:18.546722 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0926 15:09:18.547300 (+ 578us) server_negotiation.cc:299] Negotiation successful
0926 15:09:18.547445 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260926 15:09:18.548588 17016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:18.530354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:45317 (local address 127.12.34.195:36713)
0926 15:09:18.531277 (+ 923us) negotiation.cc:107] Waiting for socket to connect
0926 15:09:18.531325 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 15:09:18.533632 (+ 2307us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:09:18.534429 (+ 797us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:09:18.534444 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:09:18.534960 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:09:18.535608 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:18.535625 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:18.536785 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:18.536789 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:18.538015 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:18.538027 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:18.538833 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:18.538840 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:18.539048 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:18.539701 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:09:18.539722 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:09:18.541938 (+ 2216us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:09:18.544920 (+ 2982us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:09:18.544926 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:09:18.544936 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:09:18.545508 (+ 572us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:09:18.545945 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:09:18.545950 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:09:18.545953 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:09:18.546169 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:09:18.546719 (+ 550us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:09:18.546725 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:09:18.546997 (+ 272us) client_negotiation.cc:769] Sending connection context
0926 15:09:18.547220 (+ 223us) client_negotiation.cc:240] Negotiation successful
0926 15:09:18.547592 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":619,"thread_start_us":434,"threads_started":1}
I20260926 15:09:18.550004 17014 heartbeater.cc:347] Connected to a master server at 127.12.34.254:45317
I20260926 15:09:18.550289 17014 heartbeater.cc:464] Registering TS with master...
I20260926 15:09:18.551014 17014 heartbeater.cc:511] Master 127.12.34.254:45317 requested a full tablet report, sending...
I20260926 15:09:18.552273 16561 ts_manager.cc:194] Registered new tserver with Master: 245c8d956046409b8cc89fbaf0e9c243 (127.12.34.195:37409)
I20260926 15:09:18.553037 16561 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.34.195@KRBTEST.COM'} at 127.12.34.195:36713
I20260926 15:09:18.558348 12427 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 15:09:18.560563 16749 heartbeater.cc:503] Master 127.12.34.254:45317 was elected leader, sending a full tablet report...
I20260926 15:09:18.579039 16751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:18.562991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59834 (local address 127.12.34.254:45317)
0926 15:09:18.563220 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:18.563228 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:18.563653 (+ 425us) server_negotiation.cc:407] Connection header received
0926 15:09:18.563844 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:18.563849 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:18.563921 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:18.564023 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:09:18.565428 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:18.566140 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:18.568259 (+ 2119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:18.569032 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:18.569202 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:18.572192 (+ 2990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:09:18.572198 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:09:18.572202 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:09:18.572236 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:09:18.576050 (+ 3814us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:18.576792 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:18.576797 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:18.576800 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:18.576942 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:18.577372 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:18.577376 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:18.577380 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:18.577771 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:09:18.578003 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0926 15:09:18.578425 (+ 422us) server_negotiation.cc:299] Negotiation successful
0926 15:09:18.578685 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 15:09:18.583274 16561 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59834:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 15:09:18.590039 16561 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 15:09:18.626108 17028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:18.601938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42126 (local address 127.12.34.193:35561)
0926 15:09:18.614340 (+ 12402us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:18.614353 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:18.616784 (+ 2431us) server_negotiation.cc:407] Connection header received
0926 15:09:18.617209 (+ 425us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:18.617216 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:18.617476 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:18.617684 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:09:18.619089 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:18.620687 (+ 1598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:18.623516 (+ 2829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:18.624952 (+ 1436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:18.625120 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:18.625351 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0926 15:09:18.625606 (+ 255us) server_negotiation.cc:299] Negotiation successful
0926 15:09:18.625755 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12150,"thread_start_us":12407,"threads_started":1}
I20260926 15:09:18.626145 17026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:18.601629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.193:35561 (local address 127.0.0.1:42126)
0926 15:09:18.616612 (+ 14983us) negotiation.cc:107] Waiting for socket to connect
0926 15:09:18.616653 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 15:09:18.616883 (+ 230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:09:18.617947 (+ 1064us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:09:18.617957 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:09:18.617992 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:09:18.618412 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:18.618429 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:18.621173 (+ 2744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:18.621177 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:18.623317 (+ 2140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:18.623331 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:18.625162 (+ 1831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:18.625168 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:18.625361 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:18.625421 (+ 60us) client_negotiation.cc:769] Sending connection context
0926 15:09:18.625569 (+ 148us) client_negotiation.cc:240] Negotiation successful
0926 15:09:18.625743 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14778,"mutex_wait_us":2,"thread_start_us":15198,"threads_started":1}
I20260926 15:09:18.627431 17030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:18.602959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55280 (local address 127.12.34.194:42063)
0926 15:09:18.618317 (+ 15358us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:18.618326 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:18.619328 (+ 1002us) server_negotiation.cc:407] Connection header received
0926 15:09:18.619419 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:18.619425 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:18.619638 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:18.619810 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:09:18.620575 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:18.622128 (+ 1553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:18.624862 (+ 2734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:18.626238 (+ 1376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:18.626358 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:18.626520 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0926 15:09:18.626997 (+ 477us) server_negotiation.cc:299] Negotiation successful
0926 15:09:18.627154 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15200,"thread_start_us":15101,"threads_started":1}
I20260926 15:09:18.627454 17029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:18.601857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.194:42063 (local address 127.0.0.1:55280)
0926 15:09:18.619060 (+ 17203us) negotiation.cc:107] Waiting for socket to connect
0926 15:09:18.619078 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 15:09:18.619208 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:09:18.620033 (+ 825us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:09:18.620038 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:09:18.620063 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:09:18.620418 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:18.620425 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:18.622298 (+ 1873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:18.622304 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:18.624695 (+ 2391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:18.624710 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:18.626649 (+ 1939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:18.626654 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:18.626803 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:18.626845 (+ 42us) client_negotiation.cc:769] Sending connection context
0926 15:09:18.626941 (+ 96us) client_negotiation.cc:240] Negotiation successful
0926 15:09:18.627061 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17063,"mutex_wait_us":72,"thread_start_us":17060,"threads_started":1}
I20260926 15:09:18.628378 17031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:18.603311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57574 (local address 127.12.34.195:37409)
0926 15:09:18.617935 (+ 14624us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:18.617942 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:18.619304 (+ 1362us) server_negotiation.cc:407] Connection header received
0926 15:09:18.619576 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:18.619583 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:18.619781 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:18.619939 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:09:18.620870 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:18.622443 (+ 1573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:18.624474 (+ 2031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:18.626085 (+ 1611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:18.626201 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:18.626434 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0926 15:09:18.628035 (+ 1601us) server_negotiation.cc:299] Negotiation successful
0926 15:09:18.628151 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14422,"thread_start_us":14283,"threads_started":1}
I20260926 15:09:18.628463 17027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:18.602292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.195:37409 (local address 127.0.0.1:57574)
0926 15:09:18.619199 (+ 16907us) negotiation.cc:107] Waiting for socket to connect
0926 15:09:18.619218 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 15:09:18.619400 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:09:18.620236 (+ 836us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:09:18.620244 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:09:18.620267 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:09:18.620654 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:18.620664 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:18.622596 (+ 1932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:18.622600 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:18.624314 (+ 1714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:18.624325 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:18.627684 (+ 3359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:18.627690 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:18.627859 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:18.627896 (+ 37us) client_negotiation.cc:769] Sending connection context
0926 15:09:18.627990 (+ 94us) client_negotiation.cc:240] Negotiation successful
0926 15:09:18.628103 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16764,"mutex_wait_us":27,"thread_start_us":16166,"threads_started":1}
I20260926 15:09:18.630105 16684 tablet_service.cc:1467] Processing CreateTablet for tablet d66d5b5ef8294c7db83e139b085a78b7 (DEFAULT_TABLE table=test-table [id=fa6326d3355a4bf58cef8a0724bdc151]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:09:18.631600 16684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d66d5b5ef8294c7db83e139b085a78b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:09:18.631639 16949 tablet_service.cc:1467] Processing CreateTablet for tablet d66d5b5ef8294c7db83e139b085a78b7 (DEFAULT_TABLE table=test-table [id=fa6326d3355a4bf58cef8a0724bdc151]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:09:18.632544 16949 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d66d5b5ef8294c7db83e139b085a78b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:09:18.632192 16817 tablet_service.cc:1467] Processing CreateTablet for tablet d66d5b5ef8294c7db83e139b085a78b7 (DEFAULT_TABLE table=test-table [id=fa6326d3355a4bf58cef8a0724bdc151]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:09:18.633375 16817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d66d5b5ef8294c7db83e139b085a78b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:09:18.643472 17032 tablet_bootstrap.cc:492] T d66d5b5ef8294c7db83e139b085a78b7 P 245c8d956046409b8cc89fbaf0e9c243: Bootstrap starting.
I20260926 15:09:18.645460 17033 tablet_bootstrap.cc:492] T d66d5b5ef8294c7db83e139b085a78b7 P 8a094011d4924b2b86082363f5750607: Bootstrap starting.
I20260926 15:09:18.647007 17032 tablet_bootstrap.cc:654] T d66d5b5ef8294c7db83e139b085a78b7 P 245c8d956046409b8cc89fbaf0e9c243: Neither blocks nor log segments found. Creating new log.
I20260926 15:09:18.647171 17034 tablet_bootstrap.cc:492] T d66d5b5ef8294c7db83e139b085a78b7 P d1ee0420925e4a4a831dd047c84c0b1e: Bootstrap starting.
I20260926 15:09:18.648700 17032 log.cc:824] T d66d5b5ef8294c7db83e139b085a78b7 P 245c8d956046409b8cc89fbaf0e9c243: Log is configured to *not* fsync() on all Append() calls
I20260926 15:09:18.649187 17033 tablet_bootstrap.cc:654] T d66d5b5ef8294c7db83e139b085a78b7 P 8a094011d4924b2b86082363f5750607: Neither blocks nor log segments found. Creating new log.
I20260926 15:09:18.650223 17033 log.cc:824] T d66d5b5ef8294c7db83e139b085a78b7 P 8a094011d4924b2b86082363f5750607: Log is configured to *not* fsync() on all Append() calls
I20260926 15:09:18.650445 17034 tablet_bootstrap.cc:654] T d66d5b5ef8294c7db83e139b085a78b7 P d1ee0420925e4a4a831dd047c84c0b1e: Neither blocks nor log segments found. Creating new log.
I20260926 15:09:18.651615 17034 log.cc:824] T d66d5b5ef8294c7db83e139b085a78b7 P d1ee0420925e4a4a831dd047c84c0b1e: Log is configured to *not* fsync() on all Append() calls
I20260926 15:09:18.651682 17032 tablet_bootstrap.cc:492] T d66d5b5ef8294c7db83e139b085a78b7 P 245c8d956046409b8cc89fbaf0e9c243: No bootstrap required, opened a new log
I20260926 15:09:18.651921 17032 ts_tablet_manager.cc:1397] T d66d5b5ef8294c7db83e139b085a78b7 P 245c8d956046409b8cc89fbaf0e9c243: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260926 15:09:18.653013 17033 tablet_bootstrap.cc:492] T d66d5b5ef8294c7db83e139b085a78b7 P 8a094011d4924b2b86082363f5750607: No bootstrap required, opened a new log
I20260926 15:09:18.653288 17033 ts_tablet_manager.cc:1397] T d66d5b5ef8294c7db83e139b085a78b7 P 8a094011d4924b2b86082363f5750607: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260926 15:09:18.653923 17034 tablet_bootstrap.cc:492] T d66d5b5ef8294c7db83e139b085a78b7 P d1ee0420925e4a4a831dd047c84c0b1e: No bootstrap required, opened a new log
I20260926 15:09:18.654151 17034 ts_tablet_manager.cc:1397] T d66d5b5ef8294c7db83e139b085a78b7 P d1ee0420925e4a4a831dd047c84c0b1e: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.005s
I20260926 15:09:18.661227 17034 raft_consensus.cc:348] T d66d5b5ef8294c7db83e139b085a78b7 P d1ee0420925e4a4a831dd047c84c0b1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a094011d4924b2b86082363f5750607" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 35561 } } peers { permanent_uuid: "d1ee0420925e4a4a831dd047c84c0b1e" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 42063 } } peers { permanent_uuid: "245c8d956046409b8cc89fbaf0e9c243" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 37409 } }
I20260926 15:09:18.661577 17034 raft_consensus.cc:374] T d66d5b5ef8294c7db83e139b085a78b7 P d1ee0420925e4a4a831dd047c84c0b1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:09:18.661669 17034 raft_consensus.cc:729] T d66d5b5ef8294c7db83e139b085a78b7 P d1ee0420925e4a4a831dd047c84c0b1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1ee0420925e4a4a831dd047c84c0b1e, State: Initialized, Role: FOLLOWER
I20260926 15:09:18.662245 17034 consensus_queue.cc:260] T d66d5b5ef8294c7db83e139b085a78b7 P d1ee0420925e4a4a831dd047c84c0b1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a094011d4924b2b86082363f5750607" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 35561 } } peers { permanent_uuid: "d1ee0420925e4a4a831dd047c84c0b1e" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 42063 } } peers { permanent_uuid: "245c8d956046409b8cc89fbaf0e9c243" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 37409 } }
I20260926 15:09:18.662205 17032 raft_consensus.cc:348] T d66d5b5ef8294c7db83e139b085a78b7 P 245c8d956046409b8cc89fbaf0e9c243 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a094011d4924b2b86082363f5750607" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 35561 } } peers { permanent_uuid: "d1ee0420925e4a4a831dd047c84c0b1e" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 42063 } } peers { permanent_uuid: "245c8d956046409b8cc89fbaf0e9c243" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 37409 } }
I20260926 15:09:18.662583 17032 raft_consensus.cc:374] T d66d5b5ef8294c7db83e139b085a78b7 P 245c8d956046409b8cc89fbaf0e9c243 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:09:18.662681 17032 raft_consensus.cc:729] T d66d5b5ef8294c7db83e139b085a78b7 P 245c8d956046409b8cc89fbaf0e9c243 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 245c8d956046409b8cc89fbaf0e9c243, State: Initialized, Role: FOLLOWER
I20260926 15:09:18.663215 17032 consensus_queue.cc:260] T d66d5b5ef8294c7db83e139b085a78b7 P 245c8d956046409b8cc89fbaf0e9c243 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a094011d4924b2b86082363f5750607" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 35561 } } peers { permanent_uuid: "d1ee0420925e4a4a831dd047c84c0b1e" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 42063 } } peers { permanent_uuid: "245c8d956046409b8cc89fbaf0e9c243" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 37409 } }
I20260926 15:09:18.663462 16882 heartbeater.cc:503] Master 127.12.34.254:45317 was elected leader, sending a full tablet report...
I20260926 15:09:18.663359 17033 raft_consensus.cc:348] T d66d5b5ef8294c7db83e139b085a78b7 P 8a094011d4924b2b86082363f5750607 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a094011d4924b2b86082363f5750607" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 35561 } } peers { permanent_uuid: "d1ee0420925e4a4a831dd047c84c0b1e" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 42063 } } peers { permanent_uuid: "245c8d956046409b8cc89fbaf0e9c243" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 37409 } }
I20260926 15:09:18.663728 17033 raft_consensus.cc:374] T d66d5b5ef8294c7db83e139b085a78b7 P 8a094011d4924b2b86082363f5750607 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:09:18.663851 17033 raft_consensus.cc:729] T d66d5b5ef8294c7db83e139b085a78b7 P 8a094011d4924b2b86082363f5750607 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a094011d4924b2b86082363f5750607, State: Initialized, Role: FOLLOWER
I20260926 15:09:18.664060 17034 ts_tablet_manager.cc:1428] T d66d5b5ef8294c7db83e139b085a78b7 P d1ee0420925e4a4a831dd047c84c0b1e: Time spent starting tablet: real 0.010s user 0.007s sys 0.003s
I20260926 15:09:18.664593 17033 consensus_queue.cc:260] T d66d5b5ef8294c7db83e139b085a78b7 P 8a094011d4924b2b86082363f5750607 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a094011d4924b2b86082363f5750607" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 35561 } } peers { permanent_uuid: "d1ee0420925e4a4a831dd047c84c0b1e" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 42063 } } peers { permanent_uuid: "245c8d956046409b8cc89fbaf0e9c243" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 37409 } }
I20260926 15:09:18.664848 17014 heartbeater.cc:503] Master 127.12.34.254:45317 was elected leader, sending a full tablet report...
I20260926 15:09:18.665555 17032 ts_tablet_manager.cc:1428] T d66d5b5ef8294c7db83e139b085a78b7 P 245c8d956046409b8cc89fbaf0e9c243: Time spent starting tablet: real 0.014s user 0.014s sys 0.001s
I20260926 15:09:18.667532 17033 ts_tablet_manager.cc:1428] T d66d5b5ef8294c7db83e139b085a78b7 P 8a094011d4924b2b86082363f5750607: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
W20260926 15:09:18.779016 17015 tablet.cc:2369] T d66d5b5ef8294c7db83e139b085a78b7 P 245c8d956046409b8cc89fbaf0e9c243: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 15:09:18.782316 16883 tablet.cc:2369] T d66d5b5ef8294c7db83e139b085a78b7 P d1ee0420925e4a4a831dd047c84c0b1e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 15:09:18.782316 16750 tablet.cc:2369] T d66d5b5ef8294c7db83e139b085a78b7 P 8a094011d4924b2b86082363f5750607: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 15:09:18.850342 17040 raft_consensus.cc:482] T d66d5b5ef8294c7db83e139b085a78b7 P 8a094011d4924b2b86082363f5750607 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 15:09:18.850608 17040 raft_consensus.cc:504] T d66d5b5ef8294c7db83e139b085a78b7 P 8a094011d4924b2b86082363f5750607 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a094011d4924b2b86082363f5750607" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 35561 } } peers { permanent_uuid: "d1ee0420925e4a4a831dd047c84c0b1e" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 42063 } } peers { permanent_uuid: "245c8d956046409b8cc89fbaf0e9c243" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 37409 } }
I20260926 15:09:18.852522 17040 leader_election.cc:288] T d66d5b5ef8294c7db83e139b085a78b7 P 8a094011d4924b2b86082363f5750607 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1ee0420925e4a4a831dd047c84c0b1e (127.12.34.194:42063), 245c8d956046409b8cc89fbaf0e9c243 (127.12.34.195:37409)
I20260926 15:09:18.861917 17031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:18.853358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.193:58187 (local address 127.12.34.195:37409)
0926 15:09:18.853613 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:18.853621 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:18.854339 (+ 718us) server_negotiation.cc:407] Connection header received
0926 15:09:18.854710 (+ 371us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:18.854714 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:18.854799 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:18.854897 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:09:18.855504 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:18.856615 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:18.859544 (+ 2929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:18.860793 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:18.860957 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:18.861028 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 15:09:18.861527 (+ 499us) server_negotiation.cc:299] Negotiation successful
0926 15:09:18.861651 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260926 15:09:18.861912 17030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:18.853128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.193:59405 (local address 127.12.34.194:42063)
0926 15:09:18.853356 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:18.853365 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:18.854593 (+ 1228us) server_negotiation.cc:407] Connection header received
0926 15:09:18.854870 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:18.854877 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:18.854945 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:18.855044 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:09:18.855661 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:18.856690 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:18.859544 (+ 2854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:18.860823 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:18.860955 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:18.861028 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 15:09:18.861527 (+ 499us) server_negotiation.cc:299] Negotiation successful
0926 15:09:18.861651 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 15:09:18.862143 17042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:18.852820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.195:37409 (local address 127.12.34.193:58187)
0926 15:09:18.854225 (+ 1405us) negotiation.cc:107] Waiting for socket to connect
0926 15:09:18.854252 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0926 15:09:18.854499 (+ 247us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:09:18.854908 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:09:18.854911 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:09:18.854929 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:09:18.855348 (+ 419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:18.855362 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:18.856767 (+ 1405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:18.856774 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:18.859404 (+ 2630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:18.859413 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:18.861071 (+ 1658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:18.861098 (+ 27us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:18.861315 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:18.861395 (+ 80us) client_negotiation.cc:769] Sending connection context
0926 15:09:18.861493 (+ 98us) client_negotiation.cc:240] Negotiation successful
0926 15:09:18.861662 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1233,"mutex_wait_us":193,"thread_start_us":1375,"threads_started":1}
I20260926 15:09:18.862447 17041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:18.853036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.194:42063 (local address 127.12.34.193:59405)
0926 15:09:18.854504 (+ 1468us) negotiation.cc:107] Waiting for socket to connect
0926 15:09:18.854521 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 15:09:18.854711 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:09:18.855086 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:09:18.855090 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:09:18.855114 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:09:18.855513 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:18.855523 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:18.856813 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:18.856823 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:18.859382 (+ 2559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:18.859400 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:18.861056 (+ 1656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:18.861063 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:18.861292 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:18.861396 (+ 104us) client_negotiation.cc:769] Sending connection context
0926 15:09:18.861494 (+ 98us) client_negotiation.cc:240] Negotiation successful
0926 15:09:18.861662 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1350,"mutex_wait_us":191,"thread_start_us":751,"threads_started":1}
I20260926 15:09:18.863233 16969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d66d5b5ef8294c7db83e139b085a78b7" candidate_uuid: "8a094011d4924b2b86082363f5750607" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "245c8d956046409b8cc89fbaf0e9c243" is_pre_election: true
I20260926 15:09:18.863395 16837 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d66d5b5ef8294c7db83e139b085a78b7" candidate_uuid: "8a094011d4924b2b86082363f5750607" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1ee0420925e4a4a831dd047c84c0b1e" is_pre_election: true
I20260926 15:09:18.863840 16837 raft_consensus.cc:2454] T d66d5b5ef8294c7db83e139b085a78b7 P d1ee0420925e4a4a831dd047c84c0b1e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a094011d4924b2b86082363f5750607 in term 0.
I20260926 15:09:18.863802 16969 raft_consensus.cc:2454] T d66d5b5ef8294c7db83e139b085a78b7 P 245c8d956046409b8cc89fbaf0e9c243 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a094011d4924b2b86082363f5750607 in term 0.
I20260926 15:09:18.864472 16639 leader_election.cc:302] T d66d5b5ef8294c7db83e139b085a78b7 P 8a094011d4924b2b86082363f5750607 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a094011d4924b2b86082363f5750607, d1ee0420925e4a4a831dd047c84c0b1e; no voters:
I20260926 15:09:18.864877 17040 raft_consensus.cc:2781] T d66d5b5ef8294c7db83e139b085a78b7 P 8a094011d4924b2b86082363f5750607 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 15:09:18.865015 17040 raft_consensus.cc:482] T d66d5b5ef8294c7db83e139b085a78b7 P 8a094011d4924b2b86082363f5750607 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 15:09:18.865183 17040 raft_consensus.cc:3037] T d66d5b5ef8294c7db83e139b085a78b7 P 8a094011d4924b2b86082363f5750607 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:09:18.867286 17040 raft_consensus.cc:504] T d66d5b5ef8294c7db83e139b085a78b7 P 8a094011d4924b2b86082363f5750607 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a094011d4924b2b86082363f5750607" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 35561 } } peers { permanent_uuid: "d1ee0420925e4a4a831dd047c84c0b1e" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 42063 } } peers { permanent_uuid: "245c8d956046409b8cc89fbaf0e9c243" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 37409 } }
I20260926 15:09:18.867859 17040 leader_election.cc:288] T d66d5b5ef8294c7db83e139b085a78b7 P 8a094011d4924b2b86082363f5750607 [CANDIDATE]: Term 1 election: Requested vote from peers d1ee0420925e4a4a831dd047c84c0b1e (127.12.34.194:42063), 245c8d956046409b8cc89fbaf0e9c243 (127.12.34.195:37409)
I20260926 15:09:18.868242 16837 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d66d5b5ef8294c7db83e139b085a78b7" candidate_uuid: "8a094011d4924b2b86082363f5750607" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1ee0420925e4a4a831dd047c84c0b1e"
I20260926 15:09:18.868305 16969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d66d5b5ef8294c7db83e139b085a78b7" candidate_uuid: "8a094011d4924b2b86082363f5750607" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "245c8d956046409b8cc89fbaf0e9c243"
I20260926 15:09:18.868439 16837 raft_consensus.cc:3037] T d66d5b5ef8294c7db83e139b085a78b7 P d1ee0420925e4a4a831dd047c84c0b1e [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:09:18.868484 16969 raft_consensus.cc:3037] T d66d5b5ef8294c7db83e139b085a78b7 P 245c8d956046409b8cc89fbaf0e9c243 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:09:18.871520 16837 raft_consensus.cc:2454] T d66d5b5ef8294c7db83e139b085a78b7 P d1ee0420925e4a4a831dd047c84c0b1e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a094011d4924b2b86082363f5750607 in term 1.
I20260926 15:09:18.871536 16969 raft_consensus.cc:2454] T d66d5b5ef8294c7db83e139b085a78b7 P 245c8d956046409b8cc89fbaf0e9c243 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a094011d4924b2b86082363f5750607 in term 1.
I20260926 15:09:18.871939 16639 leader_election.cc:302] T d66d5b5ef8294c7db83e139b085a78b7 P 8a094011d4924b2b86082363f5750607 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a094011d4924b2b86082363f5750607, d1ee0420925e4a4a831dd047c84c0b1e; no voters:
I20260926 15:09:18.872203 17040 raft_consensus.cc:2781] T d66d5b5ef8294c7db83e139b085a78b7 P 8a094011d4924b2b86082363f5750607 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:09:18.872706 17040 raft_consensus.cc:686] T d66d5b5ef8294c7db83e139b085a78b7 P 8a094011d4924b2b86082363f5750607 [term 1 LEADER]: Becoming Leader. State: Replica: 8a094011d4924b2b86082363f5750607, State: Running, Role: LEADER
I20260926 15:09:18.873291 17040 consensus_queue.cc:237] T d66d5b5ef8294c7db83e139b085a78b7 P 8a094011d4924b2b86082363f5750607 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a094011d4924b2b86082363f5750607" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 35561 } } peers { permanent_uuid: "d1ee0420925e4a4a831dd047c84c0b1e" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 42063 } } peers { permanent_uuid: "245c8d956046409b8cc89fbaf0e9c243" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 37409 } }
I20260926 15:09:18.877662 16561 catalog_manager.cc:5495] T d66d5b5ef8294c7db83e139b085a78b7 P 8a094011d4924b2b86082363f5750607 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8a094011d4924b2b86082363f5750607 (127.12.34.193). New cstate: current_term: 1 leader_uuid: "8a094011d4924b2b86082363f5750607" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a094011d4924b2b86082363f5750607" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 35561 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1ee0420925e4a4a831dd047c84c0b1e" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 42063 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "245c8d956046409b8cc89fbaf0e9c243" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 37409 } health_report { overall_health: UNKNOWN } } }
I20260926 15:09:18.923911 17028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:18.918443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42142 (local address 127.12.34.193:35561)
0926 15:09:18.918651 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:18.918656 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:18.918786 (+ 130us) server_negotiation.cc:407] Connection header received
0926 15:09:18.918894 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:18.918898 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:18.918991 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:18.919120 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 15:09:18.919531 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:18.920246 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:18.921775 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:18.922245 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:18.922396 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:18.922885 (+ 489us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 15:09:18.923420 (+ 535us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 15:09:18.923536 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0926 15:09:18.923646 (+ 110us) server_negotiation.cc:299] Negotiation successful
0926 15:09:18.923711 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 15:09:18.931927 16837 raft_consensus.cc:1261] T d66d5b5ef8294c7db83e139b085a78b7 P d1ee0420925e4a4a831dd047c84c0b1e [term 1 FOLLOWER]: Refusing update from remote peer 8a094011d4924b2b86082363f5750607: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:09:18.931927 16969 raft_consensus.cc:1261] T d66d5b5ef8294c7db83e139b085a78b7 P 245c8d956046409b8cc89fbaf0e9c243 [term 1 FOLLOWER]: Refusing update from remote peer 8a094011d4924b2b86082363f5750607: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:09:18.933218 17043 consensus_queue.cc:1035] T d66d5b5ef8294c7db83e139b085a78b7 P 8a094011d4924b2b86082363f5750607 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1ee0420925e4a4a831dd047c84c0b1e" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 42063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:09:18.933601 17040 consensus_queue.cc:1035] T d66d5b5ef8294c7db83e139b085a78b7 P 8a094011d4924b2b86082363f5750607 [LEADER]: Connected to new peer: Peer: permanent_uuid: "245c8d956046409b8cc89fbaf0e9c243" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 37409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:09:18.953655 17047 mvcc.cc:205] Tried to move back new op lower bound from 7333623230174818304 to 7333623229948416000. Current Snapshot: MvccSnapshot[applied={T|T < 7333623230174818304 or (T in {7333623230174818304})}]
I20260926 15:09:18.966338 16561 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59834:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 15:09:18.966589 16561 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59834:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 15:09:18.969846 16561 catalog_manager.cc:5782] T 00000000000000000000000000000000 P affb42e14eb5434b90bc13b1815a3b90: Sending DeleteTablet for 3 replicas of tablet d66d5b5ef8294c7db83e139b085a78b7
I20260926 15:09:18.970937 16684 tablet_service.cc:1514] Processing DeleteTablet for tablet d66d5b5ef8294c7db83e139b085a78b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:09:18 UTC) from {username='kudu', principal='kudu/127.12.34.254@KRBTEST.COM'} at 127.0.0.1:42126
I20260926 15:09:18.971030 16949 tablet_service.cc:1514] Processing DeleteTablet for tablet d66d5b5ef8294c7db83e139b085a78b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:09:18 UTC) from {username='kudu', principal='kudu/127.12.34.254@KRBTEST.COM'} at 127.0.0.1:57574
I20260926 15:09:18.971040 16817 tablet_service.cc:1514] Processing DeleteTablet for tablet d66d5b5ef8294c7db83e139b085a78b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:09:18 UTC) from {username='kudu', principal='kudu/127.12.34.254@KRBTEST.COM'} at 127.0.0.1:55280
I20260926 15:09:18.976063 12427 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 16622
I20260926 15:09:18.982945 17056 tablet_replica.cc:330] T d66d5b5ef8294c7db83e139b085a78b7 P 245c8d956046409b8cc89fbaf0e9c243: stopping tablet replica
I20260926 15:09:18.983060 17057 tablet_replica.cc:330] T d66d5b5ef8294c7db83e139b085a78b7 P d1ee0420925e4a4a831dd047c84c0b1e: stopping tablet replica
I20260926 15:09:18.983371 17056 raft_consensus.cc:2229] T d66d5b5ef8294c7db83e139b085a78b7 P 245c8d956046409b8cc89fbaf0e9c243 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 15:09:18.983501 17057 raft_consensus.cc:2229] T d66d5b5ef8294c7db83e139b085a78b7 P d1ee0420925e4a4a831dd047c84c0b1e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 15:09:18.983909 17056 raft_consensus.cc:2258] T d66d5b5ef8294c7db83e139b085a78b7 P 245c8d956046409b8cc89fbaf0e9c243 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:09:18.984018 17057 raft_consensus.cc:2258] T d66d5b5ef8294c7db83e139b085a78b7 P d1ee0420925e4a4a831dd047c84c0b1e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:09:18.986229 17057 ts_tablet_manager.cc:1905] T d66d5b5ef8294c7db83e139b085a78b7 P d1ee0420925e4a4a831dd047c84c0b1e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:09:18.991905 17055 tablet_replica.cc:330] T d66d5b5ef8294c7db83e139b085a78b7 P 8a094011d4924b2b86082363f5750607: stopping tablet replica
I20260926 15:09:18.992255 17055 raft_consensus.cc:2229] T d66d5b5ef8294c7db83e139b085a78b7 P 8a094011d4924b2b86082363f5750607 [term 1 LEADER]: Raft consensus shutting down.
I20260926 15:09:18.992818 17055 raft_consensus.cc:2258] T d66d5b5ef8294c7db83e139b085a78b7 P 8a094011d4924b2b86082363f5750607 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:09:18.993023 17057 ts_tablet_manager.cc:1918] T d66d5b5ef8294c7db83e139b085a78b7 P d1ee0420925e4a4a831dd047c84c0b1e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 15:09:18.994894 17056 ts_tablet_manager.cc:1905] T d66d5b5ef8294c7db83e139b085a78b7 P 245c8d956046409b8cc89fbaf0e9c243: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:09:18.994918 17055 ts_tablet_manager.cc:1905] T d66d5b5ef8294c7db83e139b085a78b7 P 8a094011d4924b2b86082363f5750607: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 26 15:09:18 dist-test-slave-865v krb5kdc[16517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435356, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.34.193@KRBTEST.COM
I20260926 15:09:19.001614 17056 ts_tablet_manager.cc:1918] T d66d5b5ef8294c7db83e139b085a78b7 P 245c8d956046409b8cc89fbaf0e9c243: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 15:09:19.001859 17056 log.cc:1196] T d66d5b5ef8294c7db83e139b085a78b7 P 245c8d956046409b8cc89fbaf0e9c243: Deleting WAL directory at /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435318278681-12427-0/minicluster-data/ts-2/wal/wals/d66d5b5ef8294c7db83e139b085a78b7
I20260926 15:09:19.002413 17057 log.cc:1196] T d66d5b5ef8294c7db83e139b085a78b7 P d1ee0420925e4a4a831dd047c84c0b1e: Deleting WAL directory at /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435318278681-12427-0/minicluster-data/ts-1/wal/wals/d66d5b5ef8294c7db83e139b085a78b7
I20260926 15:09:19.002789 17055 ts_tablet_manager.cc:1918] T d66d5b5ef8294c7db83e139b085a78b7 P 8a094011d4924b2b86082363f5750607: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 15:09:19.002873 17056 ts_tablet_manager.cc:1939] T d66d5b5ef8294c7db83e139b085a78b7 P 245c8d956046409b8cc89fbaf0e9c243: Deleting consensus metadata
I20260926 15:09:19.002948 17055 log.cc:1196] T d66d5b5ef8294c7db83e139b085a78b7 P 8a094011d4924b2b86082363f5750607: Deleting WAL directory at /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790435318278681-12427-0/minicluster-data/ts-0/wal/wals/d66d5b5ef8294c7db83e139b085a78b7
I20260926 15:09:19.003466 17055 ts_tablet_manager.cc:1939] T d66d5b5ef8294c7db83e139b085a78b7 P 8a094011d4924b2b86082363f5750607: Deleting consensus metadata
I20260926 15:09:19.004660 16546 catalog_manager.cc:4841] TS 245c8d956046409b8cc89fbaf0e9c243 (127.12.34.195:37409): tablet d66d5b5ef8294c7db83e139b085a78b7 (table test-table [id=fa6326d3355a4bf58cef8a0724bdc151]) successfully deleted
I20260926 15:09:19.004992 16545 catalog_manager.cc:4841] TS 8a094011d4924b2b86082363f5750607 (127.12.34.193:35561): tablet d66d5b5ef8294c7db83e139b085a78b7 (table test-table [id=fa6326d3355a4bf58cef8a0724bdc151]) successfully deleted
I20260926 15:09:19.005226 17057 ts_tablet_manager.cc:1939] T d66d5b5ef8294c7db83e139b085a78b7 P d1ee0420925e4a4a831dd047c84c0b1e: Deleting consensus metadata
I20260926 15:09:19.007166 16547 catalog_manager.cc:4841] TS d1ee0420925e4a4a831dd047c84c0b1e (127.12.34.194:42063): tablet d66d5b5ef8294c7db83e139b085a78b7 (table test-table [id=fa6326d3355a4bf58cef8a0724bdc151]) successfully deleted
I20260926 15:09:19.007577 17028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:18.977481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42150 (local address 127.12.34.193:35561)
0926 15:09:18.977653 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:18.977661 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:18.984789 (+ 7128us) server_negotiation.cc:407] Connection header received
0926 15:09:18.985126 (+ 337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:18.985135 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:18.985275 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:18.985373 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:09:18.987522 (+ 2149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:18.989112 (+ 1590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:18.993240 (+ 4128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:18.994000 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:18.994125 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:18.999173 (+ 5048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:09:18.999188 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:09:18.999205 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:09:18.999254 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:09:19.004053 (+ 4799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:19.004964 (+ 911us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:19.004976 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:19.004981 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:19.005241 (+ 260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:19.005794 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:19.005799 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:19.005802 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:19.006136 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:09:19.006689 (+ 553us) server_negotiation.cc:1035] Waiting for connection context
0926 15:09:19.006958 (+ 269us) server_negotiation.cc:299] Negotiation successful
0926 15:09:19.007184 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 15:09:19.008275 16744 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:09:19.108194 12427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8uoJGv/build/asan/bin/kudu with pid 16622
I20260926 15:09:19.126091 12427 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 16755
Sep 26 15:09:19 dist-test-slave-865v krb5kdc[16517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435356, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.34.194@KRBTEST.COM
I20260926 15:09:19.141454 17030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:19.126915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55290 (local address 127.12.34.194:42063)
0926 15:09:19.127078 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:19.127082 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:19.127348 (+ 266us) server_negotiation.cc:407] Connection header received
0926 15:09:19.127510 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:19.127515 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:19.127592 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:19.127688 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:09:19.128915 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:19.129842 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:19.131083 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:19.131511 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:19.131611 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:19.135646 (+ 4035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:09:19.135657 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:09:19.135671 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:09:19.135702 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:09:19.138634 (+ 2932us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:19.139280 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:19.139296 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:19.139299 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:19.139408 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:19.139769 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:19.139776 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:19.139780 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:19.140281 (+ 501us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:09:19.140637 (+ 356us) server_negotiation.cc:1035] Waiting for connection context
0926 15:09:19.140864 (+ 227us) server_negotiation.cc:299] Negotiation successful
0926 15:09:19.141107 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 15:09:19.142139 16877 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:09:19.229548 12427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8uoJGv/build/asan/bin/kudu with pid 16755
I20260926 15:09:19.247213 12427 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 16887
Sep 26 15:09:19 dist-test-slave-865v krb5kdc[16517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435356, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.34.195@KRBTEST.COM
I20260926 15:09:19.262553 17031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:19.248065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39838 (local address 127.12.34.195:37409)
0926 15:09:19.248258 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:19.248264 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:19.248284 (+ 20us) server_negotiation.cc:407] Connection header received
0926 15:09:19.248484 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:19.248488 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:19.248582 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:19.248692 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:09:19.249989 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:19.250729 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:19.252420 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:19.252829 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:19.252922 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:19.257183 (+ 4261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:09:19.257197 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:09:19.257213 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:09:19.257249 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:09:19.259787 (+ 2538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:19.260446 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:19.260455 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:19.260459 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:19.260594 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:19.260973 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:19.260978 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:19.260986 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:19.261409 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:09:19.261695 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0926 15:09:19.262011 (+ 316us) server_negotiation.cc:299] Negotiation successful
0926 15:09:19.262224 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 15:09:19.263162 17009 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:09:19.351516 12427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8uoJGv/build/asan/bin/kudu with pid 16887
I20260926 15:09:19.370263 12427 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 16530
I20260926 15:09:19.371317 16591 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:09:19.477029 12427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8uoJGv/build/asan/bin/kudu with pid 16530
2026-09-26T15:09:19Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4249 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-26T15:09:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T15:09:19Z Disabled control of system clock
I20260926 15:09:19.543140 12427 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8uoJGv/build/asan/bin/kudu
/tmp/dist-test-task8uoJGv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790435318278681-12427-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790435318278681-12427-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790435318278681-12427-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.34.254:33533
--webserver_interface=127.12.34.254
--webserver_port=0
--builtin_ntp_servers=127.12.34.212:38681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.34.254:33533
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 15:09:19.860217 17066 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:09:19.860492 17066 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:09:19.860574 17066 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:09:19.867651 17066 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:09:19.867758 17066 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:09:19.867794 17066 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:09:19.867877 17066 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:09:19.867944 17066 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:09:19.876968 17066 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.34.212:38681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790435318278681-12427-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790435318278681-12427-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.34.254:33533
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.34.254:33533
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.34.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790435318278681-12427-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:09:19.878331 17066 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:09:19.879992 17066 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:09:19.891590 17071 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:09:19.891520 17072 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:09:19.891556 17074 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:09:19.892465 17066 server_base.cc:1034] running on GCE node
I20260926 15:09:19.893375 17066 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:09:19.894799 17066 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:09:19.896039 17066 hybrid_clock.cc:648] HybridClock initialized: now 1790435359895990 us; error 46 us; skew 500 ppm
I20260926 15:09:19.896411 17066 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 15:09:19.898667 17066 webserver.cc:460] Webserver started at http://127.12.34.254:44793/ using document root <none> and password file <none>
I20260926 15:09:19.899345 17066 fs_manager.cc:359] Metadata directory not provided
I20260926 15:09:19.899475 17066 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:09:19.899794 17066 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:09:19.902631 17066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790435318278681-12427-0/minicluster-data/master-0/data/instance:
uuid: "caea83a62e3b47e1b21e8f835519135b"
format_stamp: "Formatted at 2026-09-26 15:09:19 on dist-test-slave-865v"
server_key: "4a215c0ff130f064d1779990f2b91eef"
server_key_iv: "a95ff898d0235b373fb4c2492e7a6b7a"
server_key_version: "encryptionkey@0"
I20260926 15:09:19.903409 17066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790435318278681-12427-0/minicluster-data/master-0/wal/instance:
uuid: "caea83a62e3b47e1b21e8f835519135b"
format_stamp: "Formatted at 2026-09-26 15:09:19 on dist-test-slave-865v"
server_key: "4a215c0ff130f064d1779990f2b91eef"
server_key_iv: "a95ff898d0235b373fb4c2492e7a6b7a"
server_key_version: "encryptionkey@0"
I20260926 15:09:19.908330 17066 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 15:09:19.911818 17080 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:09:19.913144 17066 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 15:09:19.913365 17066 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790435318278681-12427-0/minicluster-data/master-0/data,/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790435318278681-12427-0/minicluster-data/master-0/wal
uuid: "caea83a62e3b47e1b21e8f835519135b"
format_stamp: "Formatted at 2026-09-26 15:09:19 on dist-test-slave-865v"
server_key: "4a215c0ff130f064d1779990f2b91eef"
server_key_iv: "a95ff898d0235b373fb4c2492e7a6b7a"
server_key_version: "encryptionkey@0"
I20260926 15:09:19.913584 17066 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790435318278681-12427-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790435318278681-12427-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790435318278681-12427-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:09:19.919054 17066 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:09:19.919770 17066 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:09:19.920055 17066 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:09:19.938915 17066 rpc_server.cc:307] RPC server started. Bound to: 127.12.34.254:33533
I20260926 15:09:19.938956 17131 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.34.254:33533 every 8 connection(s)
I20260926 15:09:19.940330 17066 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
I20260926 15:09:19.943985 17132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:09:19.949640 12427 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8uoJGv/build/asan/bin/kudu as pid 17066
I20260926 15:09:19.949839 12427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790435318278681-12427-0/minicluster-data/master-0/wal/instance
I20260926 15:09:19.950460 12427 external_mini_cluster.cc:1428] Setting key 600b7625db1ada4efb5db3bad89334c5
I20260926 15:09:19.954104 17132 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P caea83a62e3b47e1b21e8f835519135b: Bootstrap starting.
I20260926 15:09:19.957463 17132 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P caea83a62e3b47e1b21e8f835519135b: Neither blocks nor log segments found. Creating new log.
I20260926 15:09:19.958351 17135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:19.952438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37968 (local address 127.12.34.254:33533)
0926 15:09:19.953408 (+ 970us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:19.953418 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:19.953456 (+ 38us) server_negotiation.cc:407] Connection header received
0926 15:09:19.955009 (+ 1553us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:19.955028 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:19.955417 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:19.955697 (+ 280us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:09:19.956784 (+ 1087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:09:19.956790 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:09:19.956801 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:09:19.956821 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:09:19.956873 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:09:19.956888 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:09:19.957019 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0926 15:09:19.957224 (+ 205us) server_negotiation.cc:299] Negotiation successful
0926 15:09:19.957500 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":509,"thread_start_us":326,"threads_started":1}
I20260926 15:09:19.958520 17132 log.cc:824] T 00000000000000000000000000000000 P caea83a62e3b47e1b21e8f835519135b: Log is configured to *not* fsync() on all Append() calls
I20260926 15:09:19.961226 17132 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P caea83a62e3b47e1b21e8f835519135b: No bootstrap required, opened a new log
I20260926 15:09:19.968197 17132 raft_consensus.cc:348] T 00000000000000000000000000000000 P caea83a62e3b47e1b21e8f835519135b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "caea83a62e3b47e1b21e8f835519135b" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 33533 } }
I20260926 15:09:19.968531 17132 raft_consensus.cc:374] T 00000000000000000000000000000000 P caea83a62e3b47e1b21e8f835519135b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:09:19.968634 17132 raft_consensus.cc:729] T 00000000000000000000000000000000 P caea83a62e3b47e1b21e8f835519135b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: caea83a62e3b47e1b21e8f835519135b, State: Initialized, Role: FOLLOWER
I20260926 15:09:19.969199 17132 consensus_queue.cc:260] T 00000000000000000000000000000000 P caea83a62e3b47e1b21e8f835519135b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "caea83a62e3b47e1b21e8f835519135b" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 33533 } }
I20260926 15:09:19.969429 17132 raft_consensus.cc:388] T 00000000000000000000000000000000 P caea83a62e3b47e1b21e8f835519135b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 15:09:19.969568 17132 raft_consensus.cc:482] T 00000000000000000000000000000000 P caea83a62e3b47e1b21e8f835519135b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 15:09:19.969725 17132 raft_consensus.cc:3037] T 00000000000000000000000000000000 P caea83a62e3b47e1b21e8f835519135b [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:09:19.972100 17132 raft_consensus.cc:504] T 00000000000000000000000000000000 P caea83a62e3b47e1b21e8f835519135b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "caea83a62e3b47e1b21e8f835519135b" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 33533 } }
I20260926 15:09:19.972560 17132 leader_election.cc:302] T 00000000000000000000000000000000 P caea83a62e3b47e1b21e8f835519135b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: caea83a62e3b47e1b21e8f835519135b; no voters:
I20260926 15:09:19.973222 17132 leader_election.cc:288] T 00000000000000000000000000000000 P caea83a62e3b47e1b21e8f835519135b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 15:09:19.973415 17137 raft_consensus.cc:2781] T 00000000000000000000000000000000 P caea83a62e3b47e1b21e8f835519135b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:09:19.974167 17137 raft_consensus.cc:686] T 00000000000000000000000000000000 P caea83a62e3b47e1b21e8f835519135b [term 1 LEADER]: Becoming Leader. State: Replica: caea83a62e3b47e1b21e8f835519135b, State: Running, Role: LEADER
I20260926 15:09:19.974666 17137 consensus_queue.cc:237] T 00000000000000000000000000000000 P caea83a62e3b47e1b21e8f835519135b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "caea83a62e3b47e1b21e8f835519135b" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 33533 } }
I20260926 15:09:19.975203 17132 sys_catalog.cc:564] T 00000000000000000000000000000000 P caea83a62e3b47e1b21e8f835519135b [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:09:19.977919 17139 sys_catalog.cc:455] T 00000000000000000000000000000000 P caea83a62e3b47e1b21e8f835519135b [sys.catalog]: SysCatalogTable state changed. Reason: New leader caea83a62e3b47e1b21e8f835519135b. Latest consensus state: current_term: 1 leader_uuid: "caea83a62e3b47e1b21e8f835519135b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "caea83a62e3b47e1b21e8f835519135b" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 33533 } } }
I20260926 15:09:19.977896 17138 sys_catalog.cc:455] T 00000000000000000000000000000000 P caea83a62e3b47e1b21e8f835519135b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "caea83a62e3b47e1b21e8f835519135b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "caea83a62e3b47e1b21e8f835519135b" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 33533 } } }
I20260926 15:09:19.978331 17139 sys_catalog.cc:458] T 00000000000000000000000000000000 P caea83a62e3b47e1b21e8f835519135b [sys.catalog]: This master's current role is: LEADER
I20260926 15:09:19.978374 17138 sys_catalog.cc:458] T 00000000000000000000000000000000 P caea83a62e3b47e1b21e8f835519135b [sys.catalog]: This master's current role is: LEADER
I20260926 15:09:19.980823 17144 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:09:19.985932 17144 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 15:09:19.993952 17144 catalog_manager.cc:1347] Generated new cluster ID: 2046022f5aed4b7dae0bfb4059f667dc
I20260926 15:09:19.994069 17144 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 15:09:20.015044 17144 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 15:09:20.015532 17144 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:09:20.027979 17144 catalog_manager.cc:5868] T 00000000000000000000000000000000 P caea83a62e3b47e1b21e8f835519135b: Generated new TSK 0
I20260926 15:09:20.028806 17144 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 15:09:20.044682 12427 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8uoJGv/build/asan/bin/kudu
/tmp/dist-test-task8uoJGv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790435318278681-12427-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790435318278681-12427-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790435318278681-12427-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790435318278681-12427-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.34.193:0
--local_ip_for_outbound_sockets=127.12.34.193
--webserver_interface=127.12.34.193
--webserver_port=0
--tserver_master_addrs=127.12.34.254:33533
--builtin_ntp_servers=127.12.34.212:38681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 15:09:20.415828 17156 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:09:20.416095 17156 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:09:20.416181 17156 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:09:20.423586 17156 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:09:20.423709 17156 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:09:20.423823 17156 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.34.193
I20260926 15:09:20.432679 17156 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.34.212:38681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790435318278681-12427-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790435318278681-12427-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.34.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790435318278681-12427-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.34.193
--webserver_port=0
--tserver_master_addrs=127.12.34.254:33533
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.34.193
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790435318278681-12427-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:09:20.433916 17156 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:09:20.435549 17156 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:09:20.448096 17161 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:09:20.448138 17162 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:09:20.448215 17164 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:09:20.449375 17156 server_base.cc:1034] running on GCE node
I20260926 15:09:20.449927 17156 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:09:20.450821 17156 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:09:20.452028 17156 hybrid_clock.cc:648] HybridClock initialized: now 1790435360451995 us; error 27 us; skew 500 ppm
I20260926 15:09:20.452358 17156 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 15:09:20.454792 17156 webserver.cc:460] Webserver started at http://127.12.34.193:34557/ using document root <none> and password file <none>
I20260926 15:09:20.455487 17156 fs_manager.cc:359] Metadata directory not provided
I20260926 15:09:20.455571 17156 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:09:20.455921 17156 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:09:20.458947 17156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790435318278681-12427-0/minicluster-data/ts-0/data/instance:
uuid: "22d9c22620f749a2b9db3765109f2169"
format_stamp: "Formatted at 2026-09-26 15:09:20 on dist-test-slave-865v"
server_key: "e80a6de32d8e2c903b1474ec9ed95c45"
server_key_iv: "83531868346241804b01b82483779e3a"
server_key_version: "encryptionkey@0"
I20260926 15:09:20.459753 17156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790435318278681-12427-0/minicluster-data/ts-0/wal/instance:
uuid: "22d9c22620f749a2b9db3765109f2169"
format_stamp: "Formatted at 2026-09-26 15:09:20 on dist-test-slave-865v"
server_key: "e80a6de32d8e2c903b1474ec9ed95c45"
server_key_iv: "83531868346241804b01b82483779e3a"
server_key_version: "encryptionkey@0"
I20260926 15:09:20.464650 17156 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 15:09:20.468134 17170 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:09:20.469655 17156 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 15:09:20.469817 17156 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790435318278681-12427-0/minicluster-data/ts-0/data,/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790435318278681-12427-0/minicluster-data/ts-0/wal
uuid: "22d9c22620f749a2b9db3765109f2169"
format_stamp: "Formatted at 2026-09-26 15:09:20 on dist-test-slave-865v"
server_key: "e80a6de32d8e2c903b1474ec9ed95c45"
server_key_iv: "83531868346241804b01b82483779e3a"
server_key_version: "encryptionkey@0"
I20260926 15:09:20.470031 17156 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790435318278681-12427-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790435318278681-12427-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790435318278681-12427-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:09:20.475667 17156 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:09:20.476421 17156 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:09:20.476748 17156 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:09:20.477690 17156 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:09:20.478962 17156 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:09:20.479081 17156 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:09:20.479209 17156 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:09:20.479305 17156 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:09:20.512962 17156 rpc_server.cc:307] RPC server started. Bound to: 127.12.34.193:33093
I20260926 15:09:20.513051 17282 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.34.193:33093 every 8 connection(s)
I20260926 15:09:20.514240 17156 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790435318278681-12427-0/minicluster-data/ts-0/data/info.pb
I20260926 15:09:20.517609 12427 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8uoJGv/build/asan/bin/kudu as pid 17156
I20260926 15:09:20.517802 12427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790435318278681-12427-0/minicluster-data/ts-0/wal/instance
I20260926 15:09:20.518513 12427 external_mini_cluster.cc:1428] Setting key c22047c907a406ba113e5ec6b4f3766f
I20260926 15:09:20.520917 12427 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8uoJGv/build/asan/bin/kudu
/tmp/dist-test-task8uoJGv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790435318278681-12427-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790435318278681-12427-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790435318278681-12427-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790435318278681-12427-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.34.194:0
--local_ip_for_outbound_sockets=127.12.34.194
--webserver_interface=127.12.34.194
--webserver_port=0
--tserver_master_addrs=127.12.34.254:33533
--builtin_ntp_servers=127.12.34.212:38681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260926 15:09:20.522655 17285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:20.516752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.193:58811 (local address 127.12.34.254:33533)
0926 15:09:20.517218 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:20.517225 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:20.518529 (+ 1304us) server_negotiation.cc:407] Connection header received
0926 15:09:20.520359 (+ 1830us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:20.520365 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:20.520427 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:20.520548 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:09:20.521885 (+ 1337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:09:20.521890 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:09:20.521895 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:09:20.521919 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:09:20.521957 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:09:20.521968 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:09:20.522131 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0926 15:09:20.522303 (+ 172us) server_negotiation.cc:299] Negotiation successful
0926 15:09:20.522366 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":196,"threads_started":1}
I20260926 15:09:20.523509 17286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:20.517355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:33533 (local address 127.12.34.193:58811)
0926 15:09:20.518384 (+ 1029us) negotiation.cc:107] Waiting for socket to connect
0926 15:09:20.518428 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 15:09:20.520121 (+ 1693us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:09:20.520766 (+ 645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:09:20.520773 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:09:20.520856 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:09:20.521594 (+ 738us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 15:09:20.521597 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:09:20.521632 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 15:09:20.521709 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:09:20.522086 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:09:20.522104 (+ 18us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:09:20.522112 (+ 8us) client_negotiation.cc:769] Sending connection context
0926 15:09:20.522392 (+ 280us) client_negotiation.cc:240] Negotiation successful
0926 15:09:20.522627 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":351,"threads_started":1}
I20260926 15:09:20.524941 17283 heartbeater.cc:347] Connected to a master server at 127.12.34.254:33533
I20260926 15:09:20.525266 17283 heartbeater.cc:464] Registering TS with master...
I20260926 15:09:20.525990 17283 heartbeater.cc:511] Master 127.12.34.254:33533 requested a full tablet report, sending...
I20260926 15:09:20.528270 17097 ts_manager.cc:194] Registered new tserver with Master: 22d9c22620f749a2b9db3765109f2169 (127.12.34.193:33093)
W20260926 15:09:20.857067 17287 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:09:20.857437 17287 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:09:20.857592 17287 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:09:20.864857 17287 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:09:20.865008 17287 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:09:20.865218 17287 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.34.194
I20260926 15:09:20.874410 17287 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.34.212:38681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790435318278681-12427-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790435318278681-12427-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.34.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790435318278681-12427-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.34.194
--webserver_port=0
--tserver_master_addrs=127.12.34.254:33533
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.34.194
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790435318278681-12427-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:09:20.875805 17287 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:09:20.877586 17287 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:09:20.890735 17292 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:09:20.890766 17293 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:09:20.890735 17295 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:09:20.891412 17287 server_base.cc:1034] running on GCE node
I20260926 15:09:20.892040 17287 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:09:20.892963 17287 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:09:20.894197 17287 hybrid_clock.cc:648] HybridClock initialized: now 1790435360894148 us; error 71 us; skew 500 ppm
I20260926 15:09:20.894512 17287 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 15:09:20.897189 17287 webserver.cc:460] Webserver started at http://127.12.34.194:39833/ using document root <none> and password file <none>
I20260926 15:09:20.897902 17287 fs_manager.cc:359] Metadata directory not provided
I20260926 15:09:20.898032 17287 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:09:20.898367 17287 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:09:20.901412 17287 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790435318278681-12427-0/minicluster-data/ts-1/data/instance:
uuid: "7a7a765a539f4c83adb6dcf22e590fc9"
format_stamp: "Formatted at 2026-09-26 15:09:20 on dist-test-slave-865v"
server_key: "7f4350ade11a38356437fd732daa4826"
server_key_iv: "699df4b596bcd3d33af8808fbcfe92d3"
server_key_version: "encryptionkey@0"
I20260926 15:09:20.902189 17287 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790435318278681-12427-0/minicluster-data/ts-1/wal/instance:
uuid: "7a7a765a539f4c83adb6dcf22e590fc9"
format_stamp: "Formatted at 2026-09-26 15:09:20 on dist-test-slave-865v"
server_key: "7f4350ade11a38356437fd732daa4826"
server_key_iv: "699df4b596bcd3d33af8808fbcfe92d3"
server_key_version: "encryptionkey@0"
I20260926 15:09:20.907684 17287 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 15:09:20.911252 17301 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:09:20.912573 17287 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 15:09:20.912766 17287 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790435318278681-12427-0/minicluster-data/ts-1/data,/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790435318278681-12427-0/minicluster-data/ts-1/wal
uuid: "7a7a765a539f4c83adb6dcf22e590fc9"
format_stamp: "Formatted at 2026-09-26 15:09:20 on dist-test-slave-865v"
server_key: "7f4350ade11a38356437fd732daa4826"
server_key_iv: "699df4b596bcd3d33af8808fbcfe92d3"
server_key_version: "encryptionkey@0"
I20260926 15:09:20.913009 17287 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790435318278681-12427-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790435318278681-12427-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790435318278681-12427-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:09:20.918761 17287 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:09:20.919504 17287 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:09:20.919768 17287 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:09:20.920645 17287 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:09:20.922068 17287 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:09:20.922197 17287 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:09:20.922329 17287 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:09:20.922421 17287 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:09:20.959035 17287 rpc_server.cc:307] RPC server started. Bound to: 127.12.34.194:39227
I20260926 15:09:20.959124 17413 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.34.194:39227 every 8 connection(s)
I20260926 15:09:20.960255 17287 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790435318278681-12427-0/minicluster-data/ts-1/data/info.pb
I20260926 15:09:20.967864 17285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:20.962740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.194:47053 (local address 127.12.34.254:33533)
0926 15:09:20.962940 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:20.962945 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:20.964221 (+ 1276us) server_negotiation.cc:407] Connection header received
0926 15:09:20.965868 (+ 1647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:20.965872 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:20.965937 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:20.966033 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:09:20.967177 (+ 1144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:09:20.967181 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:09:20.967184 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:09:20.967199 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:09:20.967228 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:09:20.967239 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:09:20.967347 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 15:09:20.967624 (+ 277us) server_negotiation.cc:299] Negotiation successful
0926 15:09:20.967670 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260926 15:09:20.968739 17416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:20.963219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:33533 (local address 127.12.34.194:47053)
0926 15:09:20.964054 (+ 835us) negotiation.cc:107] Waiting for socket to connect
0926 15:09:20.964094 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 15:09:20.965628 (+ 1534us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:09:20.966207 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:09:20.966214 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:09:20.966296 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:09:20.966924 (+ 628us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 15:09:20.966926 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:09:20.966959 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 15:09:20.967035 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:09:20.967414 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:09:20.967423 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:09:20.967430 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 15:09:20.967682 (+ 252us) client_negotiation.cc:240] Negotiation successful
0926 15:09:20.967857 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":257,"threads_started":1}
I20260926 15:09:20.970395 12427 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8uoJGv/build/asan/bin/kudu as pid 17287
I20260926 15:09:20.970583 12427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790435318278681-12427-0/minicluster-data/ts-1/wal/instance
I20260926 15:09:20.970782 17414 heartbeater.cc:347] Connected to a master server at 127.12.34.254:33533
I20260926 15:09:20.971091 17414 heartbeater.cc:464] Registering TS with master...
I20260926 15:09:20.971122 12427 external_mini_cluster.cc:1428] Setting key 55697a87cb30121f4e1dd7590780620c
I20260926 15:09:20.971664 17414 heartbeater.cc:511] Master 127.12.34.254:33533 requested a full tablet report, sending...
I20260926 15:09:20.973181 17097 ts_manager.cc:194] Registered new tserver with Master: 7a7a765a539f4c83adb6dcf22e590fc9 (127.12.34.194:39227)
I20260926 15:09:20.973991 12427 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8uoJGv/build/asan/bin/kudu
/tmp/dist-test-task8uoJGv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790435318278681-12427-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790435318278681-12427-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790435318278681-12427-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790435318278681-12427-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.34.195:0
--local_ip_for_outbound_sockets=127.12.34.195
--webserver_interface=127.12.34.195
--webserver_port=0
--tserver_master_addrs=127.12.34.254:33533
--builtin_ntp_servers=127.12.34.212:38681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 15:09:21.283844 17417 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:09:21.284155 17417 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:09:21.284315 17417 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:09:21.291553 17417 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:09:21.291680 17417 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:09:21.291862 17417 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.34.195
I20260926 15:09:21.300712 17417 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.34.212:38681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790435318278681-12427-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790435318278681-12427-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.34.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790435318278681-12427-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.34.195
--webserver_port=0
--tserver_master_addrs=127.12.34.254:33533
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.34.195
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790435318278681-12427-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:09:21.301959 17417 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:09:21.303610 17417 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:09:21.317045 17423 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:09:21.317162 17422 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:09:21.317068 17425 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:09:21.317958 17417 server_base.cc:1034] running on GCE node
I20260926 15:09:21.318645 17417 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:09:21.319672 17417 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:09:21.320924 17417 hybrid_clock.cc:648] HybridClock initialized: now 1790435361320876 us; error 37 us; skew 500 ppm
I20260926 15:09:21.321337 17417 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 15:09:21.323660 17417 webserver.cc:460] Webserver started at http://127.12.34.195:46753/ using document root <none> and password file <none>
I20260926 15:09:21.324354 17417 fs_manager.cc:359] Metadata directory not provided
I20260926 15:09:21.324436 17417 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:09:21.324790 17417 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:09:21.327618 17417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790435318278681-12427-0/minicluster-data/ts-2/data/instance:
uuid: "0a2b041ea3de444589f1c286ada6d129"
format_stamp: "Formatted at 2026-09-26 15:09:21 on dist-test-slave-865v"
server_key: "af0ca6be876699d0b348d794d93cb6de"
server_key_iv: "d4f27c50950b77fda8399252cd4549c9"
server_key_version: "encryptionkey@0"
I20260926 15:09:21.328413 17417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790435318278681-12427-0/minicluster-data/ts-2/wal/instance:
uuid: "0a2b041ea3de444589f1c286ada6d129"
format_stamp: "Formatted at 2026-09-26 15:09:21 on dist-test-slave-865v"
server_key: "af0ca6be876699d0b348d794d93cb6de"
server_key_iv: "d4f27c50950b77fda8399252cd4549c9"
server_key_version: "encryptionkey@0"
I20260926 15:09:21.333389 17417 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260926 15:09:21.336668 17431 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:09:21.338047 17417 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 15:09:21.338230 17417 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790435318278681-12427-0/minicluster-data/ts-2/data,/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790435318278681-12427-0/minicluster-data/ts-2/wal
uuid: "0a2b041ea3de444589f1c286ada6d129"
format_stamp: "Formatted at 2026-09-26 15:09:21 on dist-test-slave-865v"
server_key: "af0ca6be876699d0b348d794d93cb6de"
server_key_iv: "d4f27c50950b77fda8399252cd4549c9"
server_key_version: "encryptionkey@0"
I20260926 15:09:21.338457 17417 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790435318278681-12427-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790435318278681-12427-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790435318278681-12427-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:09:21.343853 17417 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:09:21.344599 17417 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:09:21.344856 17417 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:09:21.345777 17417 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:09:21.347028 17417 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:09:21.347150 17417 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:09:21.347291 17417 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:09:21.347388 17417 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:09:21.379411 17417 rpc_server.cc:307] RPC server started. Bound to: 127.12.34.195:38283
I20260926 15:09:21.379473 17543 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.34.195:38283 every 8 connection(s)
I20260926 15:09:21.380795 17417 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790435318278681-12427-0/minicluster-data/ts-2/data/info.pb
I20260926 15:09:21.381875 12427 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8uoJGv/build/asan/bin/kudu as pid 17417
I20260926 15:09:21.382046 12427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790435318278681-12427-0/minicluster-data/ts-2/wal/instance
I20260926 15:09:21.382462 12427 external_mini_cluster.cc:1428] Setting key 85268c94ad4cb3fa9962fdbef3169cf4
I20260926 15:09:21.388926 17285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:21.383279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.195:58059 (local address 127.12.34.254:33533)
0926 15:09:21.383464 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:21.383468 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:21.385120 (+ 1652us) server_negotiation.cc:407] Connection header received
0926 15:09:21.386794 (+ 1674us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:21.386799 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:21.386865 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:21.386988 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:09:21.388225 (+ 1237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:09:21.388230 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:09:21.388235 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:09:21.388252 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:09:21.388292 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:09:21.388303 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:09:21.388427 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0926 15:09:21.388617 (+ 190us) server_negotiation.cc:299] Negotiation successful
0926 15:09:21.388678 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 15:09:21.389788 17546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:21.383964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:33533 (local address 127.12.34.195:58059)
0926 15:09:21.384964 (+ 1000us) negotiation.cc:107] Waiting for socket to connect
0926 15:09:21.385006 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 15:09:21.386556 (+ 1550us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:09:21.387184 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:09:21.387191 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:09:21.387287 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:09:21.387962 (+ 675us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 15:09:21.387965 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:09:21.387999 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 15:09:21.388073 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:09:21.388395 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:09:21.388404 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:09:21.388414 (+ 10us) client_negotiation.cc:769] Sending connection context
0926 15:09:21.388682 (+ 268us) client_negotiation.cc:240] Negotiation successful
0926 15:09:21.388901 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":325,"threads_started":1}
I20260926 15:09:21.390978 17544 heartbeater.cc:347] Connected to a master server at 127.12.34.254:33533
I20260926 15:09:21.391249 17544 heartbeater.cc:464] Registering TS with master...
I20260926 15:09:21.391793 17544 heartbeater.cc:511] Master 127.12.34.254:33533 requested a full tablet report, sending...
I20260926 15:09:21.392971 17097 ts_manager.cc:194] Registered new tserver with Master: 0a2b041ea3de444589f1c286ada6d129 (127.12.34.195:38283)
I20260926 15:09:21.397588 12427 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 15:09:21.404191 17285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:21.401786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37980 (local address 127.12.34.254:33533)
0926 15:09:21.401946 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:21.401950 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:21.402435 (+ 485us) server_negotiation.cc:407] Connection header received
0926 15:09:21.402530 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:21.402533 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:21.402579 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:21.402680 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:09:21.403565 (+ 885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:09:21.403579 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:09:21.403582 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:09:21.403598 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:09:21.403630 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:09:21.403642 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:09:21.403749 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 15:09:21.403823 (+ 74us) server_negotiation.cc:299] Negotiation successful
0926 15:09:21.403867 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 15:09:21.407799 17097 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37980:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 15:09:21.414462 17097 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 15:09:21.445603 17557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:21.425434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57146 (local address 127.12.34.194:39227)
0926 15:09:21.442671 (+ 17237us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:21.442677 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:21.442701 (+ 24us) server_negotiation.cc:407] Connection header received
0926 15:09:21.442754 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:21.442760 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:21.442996 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:21.443165 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:09:21.444696 (+ 1531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:09:21.444707 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:09:21.444721 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:09:21.444741 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:09:21.444798 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:09:21.444824 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:09:21.444980 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0926 15:09:21.445149 (+ 169us) server_negotiation.cc:299] Negotiation successful
0926 15:09:21.445251 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17083,"thread_start_us":16968,"threads_started":1}
I20260926 15:09:21.445680 17558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:21.425314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.194:39227 (local address 127.0.0.1:57146)
0926 15:09:21.436838 (+ 11524us) negotiation.cc:107] Waiting for socket to connect
0926 15:09:21.436881 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 15:09:21.437006 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:09:21.443181 (+ 6175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:09:21.443188 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:09:21.443277 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:09:21.444413 (+ 1136us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 15:09:21.444417 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:09:21.444447 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 15:09:21.444523 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:09:21.444935 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:09:21.444948 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:09:21.444956 (+ 8us) client_negotiation.cc:769] Sending connection context
0926 15:09:21.445170 (+ 214us) client_negotiation.cc:240] Negotiation successful
0926 15:09:21.445252 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11287,"thread_start_us":15506,"threads_started":1}
I20260926 15:09:21.447124 17560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:21.426359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45052 (local address 127.12.34.193:33093)
0926 15:09:21.443323 (+ 16964us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:21.443331 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:21.443368 (+ 37us) server_negotiation.cc:407] Connection header received
0926 15:09:21.443436 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:21.443443 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:21.443617 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:21.443751 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:09:21.446195 (+ 2444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:09:21.446204 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:09:21.446218 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:09:21.446242 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:09:21.446305 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:09:21.446332 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:09:21.446496 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0926 15:09:21.446726 (+ 230us) server_negotiation.cc:299] Negotiation successful
0926 15:09:21.446824 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16812,"thread_start_us":18004,"threads_started":1}
I20260926 15:09:21.447163 17556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:21.425739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.193:33093 (local address 127.0.0.1:45052)
0926 15:09:21.440600 (+ 14861us) negotiation.cc:107] Waiting for socket to connect
0926 15:09:21.440629 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0926 15:09:21.440741 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:09:21.444305 (+ 3564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:09:21.444312 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:09:21.444342 (+ 30us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:09:21.445970 (+ 1628us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 15:09:21.445974 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:09:21.445990 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 15:09:21.446029 (+ 39us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:09:21.446556 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:09:21.446564 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:09:21.446571 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 15:09:21.446719 (+ 148us) client_negotiation.cc:240] Negotiation successful
0926 15:09:21.446771 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14691,"thread_start_us":10760,"threads_started":1}
I20260926 15:09:21.447839 17559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:21.426091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.195:38283 (local address 127.0.0.1:45866)
0926 15:09:21.441742 (+ 15651us) negotiation.cc:107] Waiting for socket to connect
0926 15:09:21.442390 (+ 648us) client_negotiation.cc:174] Beginning negotiation
0926 15:09:21.442519 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:09:21.443680 (+ 1161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:09:21.443687 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:09:21.443715 (+ 28us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:09:21.446630 (+ 2915us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 15:09:21.446635 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:09:21.446653 (+ 18us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 15:09:21.446700 (+ 47us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:09:21.447198 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:09:21.447382 (+ 184us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:09:21.447392 (+ 10us) client_negotiation.cc:769] Sending connection context
0926 15:09:21.447533 (+ 141us) client_negotiation.cc:240] Negotiation successful
0926 15:09:21.447581 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15534,"thread_start_us":15407,"threads_started":1}
I20260926 15:09:21.448601 17561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:21.426708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45866 (local address 127.12.34.195:38283)
0926 15:09:21.442316 (+ 15608us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:21.442330 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:21.442646 (+ 316us) server_negotiation.cc:407] Connection header received
0926 15:09:21.442755 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:21.442766 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:21.443051 (+ 285us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:21.443196 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:09:21.446859 (+ 3663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:09:21.446870 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:09:21.446884 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:09:21.446901 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:09:21.446963 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:09:21.446984 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:09:21.447142 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0926 15:09:21.448111 (+ 969us) server_negotiation.cc:299] Negotiation successful
0926 15:09:21.448219 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15387,"thread_start_us":15215,"threads_started":1}
I20260926 15:09:21.449558 17349 tablet_service.cc:1467] Processing CreateTablet for tablet bb3de9c027d947189e1264533feb1018 (DEFAULT_TABLE table=test-table [id=6efca336ccd04fdfbded71d8f467fec7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:09:21.450727 17349 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb3de9c027d947189e1264533feb1018. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:09:21.450700 17218 tablet_service.cc:1467] Processing CreateTablet for tablet bb3de9c027d947189e1264533feb1018 (DEFAULT_TABLE table=test-table [id=6efca336ccd04fdfbded71d8f467fec7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:09:21.451925 17218 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb3de9c027d947189e1264533feb1018. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:09:21.451758 17479 tablet_service.cc:1467] Processing CreateTablet for tablet bb3de9c027d947189e1264533feb1018 (DEFAULT_TABLE table=test-table [id=6efca336ccd04fdfbded71d8f467fec7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:09:21.452976 17479 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb3de9c027d947189e1264533feb1018. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:09:21.461529 17562 tablet_bootstrap.cc:492] T bb3de9c027d947189e1264533feb1018 P 7a7a765a539f4c83adb6dcf22e590fc9: Bootstrap starting.
I20260926 15:09:21.464448 17562 tablet_bootstrap.cc:654] T bb3de9c027d947189e1264533feb1018 P 7a7a765a539f4c83adb6dcf22e590fc9: Neither blocks nor log segments found. Creating new log.
I20260926 15:09:21.465735 17562 log.cc:824] T bb3de9c027d947189e1264533feb1018 P 7a7a765a539f4c83adb6dcf22e590fc9: Log is configured to *not* fsync() on all Append() calls
I20260926 15:09:21.467978 17563 tablet_bootstrap.cc:492] T bb3de9c027d947189e1264533feb1018 P 22d9c22620f749a2b9db3765109f2169: Bootstrap starting.
I20260926 15:09:21.468573 17562 tablet_bootstrap.cc:492] T bb3de9c027d947189e1264533feb1018 P 7a7a765a539f4c83adb6dcf22e590fc9: No bootstrap required, opened a new log
I20260926 15:09:21.468813 17562 ts_tablet_manager.cc:1397] T bb3de9c027d947189e1264533feb1018 P 7a7a765a539f4c83adb6dcf22e590fc9: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260926 15:09:21.469254 17564 tablet_bootstrap.cc:492] T bb3de9c027d947189e1264533feb1018 P 0a2b041ea3de444589f1c286ada6d129: Bootstrap starting.
I20260926 15:09:21.471392 17563 tablet_bootstrap.cc:654] T bb3de9c027d947189e1264533feb1018 P 22d9c22620f749a2b9db3765109f2169: Neither blocks nor log segments found. Creating new log.
I20260926 15:09:21.472476 17564 tablet_bootstrap.cc:654] T bb3de9c027d947189e1264533feb1018 P 0a2b041ea3de444589f1c286ada6d129: Neither blocks nor log segments found. Creating new log.
I20260926 15:09:21.472574 17563 log.cc:824] T bb3de9c027d947189e1264533feb1018 P 22d9c22620f749a2b9db3765109f2169: Log is configured to *not* fsync() on all Append() calls
I20260926 15:09:21.473680 17564 log.cc:824] T bb3de9c027d947189e1264533feb1018 P 0a2b041ea3de444589f1c286ada6d129: Log is configured to *not* fsync() on all Append() calls
I20260926 15:09:21.475634 17563 tablet_bootstrap.cc:492] T bb3de9c027d947189e1264533feb1018 P 22d9c22620f749a2b9db3765109f2169: No bootstrap required, opened a new log
I20260926 15:09:21.475883 17563 ts_tablet_manager.cc:1397] T bb3de9c027d947189e1264533feb1018 P 22d9c22620f749a2b9db3765109f2169: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260926 15:09:21.476568 17564 tablet_bootstrap.cc:492] T bb3de9c027d947189e1264533feb1018 P 0a2b041ea3de444589f1c286ada6d129: No bootstrap required, opened a new log
I20260926 15:09:21.476801 17564 ts_tablet_manager.cc:1397] T bb3de9c027d947189e1264533feb1018 P 0a2b041ea3de444589f1c286ada6d129: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260926 15:09:21.477310 17562 raft_consensus.cc:348] T bb3de9c027d947189e1264533feb1018 P 7a7a765a539f4c83adb6dcf22e590fc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a7a765a539f4c83adb6dcf22e590fc9" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 39227 } } peers { permanent_uuid: "22d9c22620f749a2b9db3765109f2169" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 33093 } } peers { permanent_uuid: "0a2b041ea3de444589f1c286ada6d129" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 38283 } }
I20260926 15:09:21.477653 17562 raft_consensus.cc:374] T bb3de9c027d947189e1264533feb1018 P 7a7a765a539f4c83adb6dcf22e590fc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:09:21.477744 17562 raft_consensus.cc:729] T bb3de9c027d947189e1264533feb1018 P 7a7a765a539f4c83adb6dcf22e590fc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a7a765a539f4c83adb6dcf22e590fc9, State: Initialized, Role: FOLLOWER
I20260926 15:09:21.478389 17562 consensus_queue.cc:260] T bb3de9c027d947189e1264533feb1018 P 7a7a765a539f4c83adb6dcf22e590fc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a7a765a539f4c83adb6dcf22e590fc9" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 39227 } } peers { permanent_uuid: "22d9c22620f749a2b9db3765109f2169" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 33093 } } peers { permanent_uuid: "0a2b041ea3de444589f1c286ada6d129" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 38283 } }
I20260926 15:09:21.481724 17414 heartbeater.cc:503] Master 127.12.34.254:33533 was elected leader, sending a full tablet report...
I20260926 15:09:21.482403 17562 ts_tablet_manager.cc:1428] T bb3de9c027d947189e1264533feb1018 P 7a7a765a539f4c83adb6dcf22e590fc9: Time spent starting tablet: real 0.013s user 0.004s sys 0.010s
I20260926 15:09:21.486191 17563 raft_consensus.cc:348] T bb3de9c027d947189e1264533feb1018 P 22d9c22620f749a2b9db3765109f2169 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a7a765a539f4c83adb6dcf22e590fc9" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 39227 } } peers { permanent_uuid: "22d9c22620f749a2b9db3765109f2169" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 33093 } } peers { permanent_uuid: "0a2b041ea3de444589f1c286ada6d129" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 38283 } }
I20260926 15:09:21.486601 17563 raft_consensus.cc:374] T bb3de9c027d947189e1264533feb1018 P 22d9c22620f749a2b9db3765109f2169 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:09:21.486683 17563 raft_consensus.cc:729] T bb3de9c027d947189e1264533feb1018 P 22d9c22620f749a2b9db3765109f2169 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22d9c22620f749a2b9db3765109f2169, State: Initialized, Role: FOLLOWER
I20260926 15:09:21.487298 17563 consensus_queue.cc:260] T bb3de9c027d947189e1264533feb1018 P 22d9c22620f749a2b9db3765109f2169 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a7a765a539f4c83adb6dcf22e590fc9" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 39227 } } peers { permanent_uuid: "22d9c22620f749a2b9db3765109f2169" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 33093 } } peers { permanent_uuid: "0a2b041ea3de444589f1c286ada6d129" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 38283 } }
I20260926 15:09:21.487229 17564 raft_consensus.cc:348] T bb3de9c027d947189e1264533feb1018 P 0a2b041ea3de444589f1c286ada6d129 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a7a765a539f4c83adb6dcf22e590fc9" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 39227 } } peers { permanent_uuid: "22d9c22620f749a2b9db3765109f2169" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 33093 } } peers { permanent_uuid: "0a2b041ea3de444589f1c286ada6d129" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 38283 } }
I20260926 15:09:21.487632 17564 raft_consensus.cc:374] T bb3de9c027d947189e1264533feb1018 P 0a2b041ea3de444589f1c286ada6d129 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:09:21.487748 17564 raft_consensus.cc:729] T bb3de9c027d947189e1264533feb1018 P 0a2b041ea3de444589f1c286ada6d129 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a2b041ea3de444589f1c286ada6d129, State: Initialized, Role: FOLLOWER
I20260926 15:09:21.488338 17283 heartbeater.cc:503] Master 127.12.34.254:33533 was elected leader, sending a full tablet report...
I20260926 15:09:21.488322 17564 consensus_queue.cc:260] T bb3de9c027d947189e1264533feb1018 P 0a2b041ea3de444589f1c286ada6d129 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a7a765a539f4c83adb6dcf22e590fc9" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 39227 } } peers { permanent_uuid: "22d9c22620f749a2b9db3765109f2169" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 33093 } } peers { permanent_uuid: "0a2b041ea3de444589f1c286ada6d129" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 38283 } }
I20260926 15:09:21.489001 17563 ts_tablet_manager.cc:1428] T bb3de9c027d947189e1264533feb1018 P 22d9c22620f749a2b9db3765109f2169: Time spent starting tablet: real 0.013s user 0.009s sys 0.005s
I20260926 15:09:21.493721 17544 heartbeater.cc:503] Master 127.12.34.254:33533 was elected leader, sending a full tablet report...
I20260926 15:09:21.494315 17564 ts_tablet_manager.cc:1428] T bb3de9c027d947189e1264533feb1018 P 0a2b041ea3de444589f1c286ada6d129: Time spent starting tablet: real 0.017s user 0.013s sys 0.004s
W20260926 15:09:21.516616 17284 tablet.cc:2369] T bb3de9c027d947189e1264533feb1018 P 22d9c22620f749a2b9db3765109f2169: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 15:09:21.631312 17570 raft_consensus.cc:482] T bb3de9c027d947189e1264533feb1018 P 0a2b041ea3de444589f1c286ada6d129 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 15:09:21.631594 17570 raft_consensus.cc:504] T bb3de9c027d947189e1264533feb1018 P 0a2b041ea3de444589f1c286ada6d129 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a7a765a539f4c83adb6dcf22e590fc9" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 39227 } } peers { permanent_uuid: "22d9c22620f749a2b9db3765109f2169" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 33093 } } peers { permanent_uuid: "0a2b041ea3de444589f1c286ada6d129" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 38283 } }
W20260926 15:09:21.632395 17545 tablet.cc:2369] T bb3de9c027d947189e1264533feb1018 P 0a2b041ea3de444589f1c286ada6d129: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 15:09:21.633620 17570 leader_election.cc:288] T bb3de9c027d947189e1264533feb1018 P 0a2b041ea3de444589f1c286ada6d129 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a7a765a539f4c83adb6dcf22e590fc9 (127.12.34.194:39227), 22d9c22620f749a2b9db3765109f2169 (127.12.34.193:33093)
I20260926 15:09:21.636406 17557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:21.633938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.195:60213 (local address 127.12.34.194:39227)
0926 15:09:21.634111 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:21.634116 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:21.634534 (+ 418us) server_negotiation.cc:407] Connection header received
0926 15:09:21.634590 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:21.634593 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:21.634637 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:21.634741 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:09:21.635783 (+ 1042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:09:21.635791 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:09:21.635797 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:09:21.635819 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:09:21.635858 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:09:21.635896 (+ 38us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:09:21.636025 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0926 15:09:21.636122 (+ 97us) server_negotiation.cc:299] Negotiation successful
0926 15:09:21.636179 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 15:09:21.636454 17546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:21.633932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.194:39227 (local address 127.12.34.195:60213)
0926 15:09:21.634254 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0926 15:09:21.634269 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 15:09:21.634412 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:09:21.634781 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:09:21.634785 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:09:21.634808 (+ 23us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:09:21.635571 (+ 763us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 15:09:21.635575 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:09:21.635591 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 15:09:21.635633 (+ 42us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:09:21.636009 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:09:21.636017 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:09:21.636025 (+ 8us) client_negotiation.cc:769] Sending connection context
0926 15:09:21.636128 (+ 103us) client_negotiation.cc:240] Negotiation successful
0926 15:09:21.636176 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230}
I20260926 15:09:21.636854 17560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:21.634260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.195:43119 (local address 127.12.34.193:33093)
0926 15:09:21.634457 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:21.634465 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:21.634966 (+ 501us) server_negotiation.cc:407] Connection header received
0926 15:09:21.635060 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:21.635067 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:21.635126 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:21.635206 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:09:21.636139 (+ 933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:09:21.636145 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:09:21.636150 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:09:21.636168 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:09:21.636201 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:09:21.636214 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:09:21.636296 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 15:09:21.636550 (+ 254us) server_negotiation.cc:299] Negotiation successful
0926 15:09:21.636611 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260926 15:09:21.637353 17369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb3de9c027d947189e1264533feb1018" candidate_uuid: "0a2b041ea3de444589f1c286ada6d129" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a7a765a539f4c83adb6dcf22e590fc9" is_pre_election: true
I20260926 15:09:21.637478 17571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:21.634116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.193:33093 (local address 127.12.34.195:43119)
0926 15:09:21.634867 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0926 15:09:21.634881 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 15:09:21.634966 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:09:21.635260 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:09:21.635265 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:09:21.635288 (+ 23us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:09:21.635929 (+ 641us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 15:09:21.635932 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:09:21.635947 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 15:09:21.635986 (+ 39us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:09:21.636374 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:09:21.636378 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:09:21.636382 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 15:09:21.637111 (+ 729us) client_negotiation.cc:240] Negotiation successful
0926 15:09:21.637157 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":642,"thread_start_us":439,"threads_started":1}
I20260926 15:09:21.637785 17369 raft_consensus.cc:2454] T bb3de9c027d947189e1264533feb1018 P 7a7a765a539f4c83adb6dcf22e590fc9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2b041ea3de444589f1c286ada6d129 in term 0.
I20260926 15:09:21.638411 17434 leader_election.cc:302] T bb3de9c027d947189e1264533feb1018 P 0a2b041ea3de444589f1c286ada6d129 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2b041ea3de444589f1c286ada6d129, 7a7a765a539f4c83adb6dcf22e590fc9; no voters:
I20260926 15:09:21.638516 17238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb3de9c027d947189e1264533feb1018" candidate_uuid: "0a2b041ea3de444589f1c286ada6d129" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22d9c22620f749a2b9db3765109f2169" is_pre_election: true
I20260926 15:09:21.638803 17570 raft_consensus.cc:2781] T bb3de9c027d947189e1264533feb1018 P 0a2b041ea3de444589f1c286ada6d129 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 15:09:21.638960 17570 raft_consensus.cc:482] T bb3de9c027d947189e1264533feb1018 P 0a2b041ea3de444589f1c286ada6d129 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 15:09:21.639075 17570 raft_consensus.cc:3037] T bb3de9c027d947189e1264533feb1018 P 0a2b041ea3de444589f1c286ada6d129 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:09:21.638986 17238 raft_consensus.cc:2454] T bb3de9c027d947189e1264533feb1018 P 22d9c22620f749a2b9db3765109f2169 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a2b041ea3de444589f1c286ada6d129 in term 0.
I20260926 15:09:21.641227 17570 raft_consensus.cc:504] T bb3de9c027d947189e1264533feb1018 P 0a2b041ea3de444589f1c286ada6d129 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a7a765a539f4c83adb6dcf22e590fc9" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 39227 } } peers { permanent_uuid: "22d9c22620f749a2b9db3765109f2169" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 33093 } } peers { permanent_uuid: "0a2b041ea3de444589f1c286ada6d129" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 38283 } }
I20260926 15:09:21.641780 17570 leader_election.cc:288] T bb3de9c027d947189e1264533feb1018 P 0a2b041ea3de444589f1c286ada6d129 [CANDIDATE]: Term 1 election: Requested vote from peers 7a7a765a539f4c83adb6dcf22e590fc9 (127.12.34.194:39227), 22d9c22620f749a2b9db3765109f2169 (127.12.34.193:33093)
I20260926 15:09:21.642210 17369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb3de9c027d947189e1264533feb1018" candidate_uuid: "0a2b041ea3de444589f1c286ada6d129" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a7a765a539f4c83adb6dcf22e590fc9"
I20260926 15:09:21.642220 17238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb3de9c027d947189e1264533feb1018" candidate_uuid: "0a2b041ea3de444589f1c286ada6d129" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22d9c22620f749a2b9db3765109f2169"
I20260926 15:09:21.642412 17238 raft_consensus.cc:3037] T bb3de9c027d947189e1264533feb1018 P 22d9c22620f749a2b9db3765109f2169 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:09:21.642434 17369 raft_consensus.cc:3037] T bb3de9c027d947189e1264533feb1018 P 7a7a765a539f4c83adb6dcf22e590fc9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:09:21.644806 17238 raft_consensus.cc:2454] T bb3de9c027d947189e1264533feb1018 P 22d9c22620f749a2b9db3765109f2169 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a2b041ea3de444589f1c286ada6d129 in term 1.
I20260926 15:09:21.644822 17369 raft_consensus.cc:2454] T bb3de9c027d947189e1264533feb1018 P 7a7a765a539f4c83adb6dcf22e590fc9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a2b041ea3de444589f1c286ada6d129 in term 1.
I20260926 15:09:21.645217 17433 leader_election.cc:302] T bb3de9c027d947189e1264533feb1018 P 0a2b041ea3de444589f1c286ada6d129 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2b041ea3de444589f1c286ada6d129, 22d9c22620f749a2b9db3765109f2169; no voters:
I20260926 15:09:21.645522 17570 raft_consensus.cc:2781] T bb3de9c027d947189e1264533feb1018 P 0a2b041ea3de444589f1c286ada6d129 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:09:21.645968 17570 raft_consensus.cc:686] T bb3de9c027d947189e1264533feb1018 P 0a2b041ea3de444589f1c286ada6d129 [term 1 LEADER]: Becoming Leader. State: Replica: 0a2b041ea3de444589f1c286ada6d129, State: Running, Role: LEADER
I20260926 15:09:21.646394 17570 consensus_queue.cc:237] T bb3de9c027d947189e1264533feb1018 P 0a2b041ea3de444589f1c286ada6d129 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a7a765a539f4c83adb6dcf22e590fc9" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 39227 } } peers { permanent_uuid: "22d9c22620f749a2b9db3765109f2169" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 33093 } } peers { permanent_uuid: "0a2b041ea3de444589f1c286ada6d129" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 38283 } }
I20260926 15:09:21.650844 17097 catalog_manager.cc:5495] T bb3de9c027d947189e1264533feb1018 P 0a2b041ea3de444589f1c286ada6d129 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0a2b041ea3de444589f1c286ada6d129 (127.12.34.195). New cstate: current_term: 1 leader_uuid: "0a2b041ea3de444589f1c286ada6d129" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a7a765a539f4c83adb6dcf22e590fc9" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 39227 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22d9c22620f749a2b9db3765109f2169" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 33093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a2b041ea3de444589f1c286ada6d129" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 38283 } health_report { overall_health: HEALTHY } } }
I20260926 15:09:21.691145 17561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:21.689301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45878 (local address 127.12.34.195:38283)
0926 15:09:21.689459 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:21.689463 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:21.689481 (+ 18us) server_negotiation.cc:407] Connection header received
0926 15:09:21.689586 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:21.689589 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:21.689664 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:21.689747 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:09:21.690615 (+ 868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:09:21.690619 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:09:21.690622 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:09:21.690638 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:09:21.690674 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:09:21.690685 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:09:21.690782 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 15:09:21.690918 (+ 136us) server_negotiation.cc:299] Negotiation successful
0926 15:09:21.690960 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 15:09:21.698923 17369 raft_consensus.cc:1261] T bb3de9c027d947189e1264533feb1018 P 7a7a765a539f4c83adb6dcf22e590fc9 [term 1 FOLLOWER]: Refusing update from remote peer 0a2b041ea3de444589f1c286ada6d129: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:09:21.698925 17238 raft_consensus.cc:1261] T bb3de9c027d947189e1264533feb1018 P 22d9c22620f749a2b9db3765109f2169 [term 1 FOLLOWER]: Refusing update from remote peer 0a2b041ea3de444589f1c286ada6d129: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:09:21.699858 17572 consensus_queue.cc:1035] T bb3de9c027d947189e1264533feb1018 P 0a2b041ea3de444589f1c286ada6d129 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a7a765a539f4c83adb6dcf22e590fc9" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 39227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:09:21.700179 17570 consensus_queue.cc:1035] T bb3de9c027d947189e1264533feb1018 P 0a2b041ea3de444589f1c286ada6d129 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22d9c22620f749a2b9db3765109f2169" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 33093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:09:21.710132 17575 mvcc.cc:205] Tried to move back new op lower bound from 7333623241509105664 to 7333623241307140096. Current Snapshot: MvccSnapshot[applied={T|T < 7333623241509105664}]
I20260926 15:09:21.711392 17576 mvcc.cc:205] Tried to move back new op lower bound from 7333623241509105664 to 7333623241307140096. Current Snapshot: MvccSnapshot[applied={T|T < 7333623241509105664}]
I20260926 15:09:21.725358 17096 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37980:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 15:09:21.725703 17096 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37980:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 15:09:21.729169 17096 catalog_manager.cc:5782] T 00000000000000000000000000000000 P caea83a62e3b47e1b21e8f835519135b: Sending DeleteTablet for 3 replicas of tablet bb3de9c027d947189e1264533feb1018
I20260926 15:09:21.730201 17349 tablet_service.cc:1514] Processing DeleteTablet for tablet bb3de9c027d947189e1264533feb1018 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:09:21 UTC) from {username='slave'} at 127.0.0.1:57146
I20260926 15:09:21.730213 17218 tablet_service.cc:1514] Processing DeleteTablet for tablet bb3de9c027d947189e1264533feb1018 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:09:21 UTC) from {username='slave'} at 127.0.0.1:45052
I20260926 15:09:21.730415 17479 tablet_service.cc:1514] Processing DeleteTablet for tablet bb3de9c027d947189e1264533feb1018 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:09:21 UTC) from {username='slave'} at 127.0.0.1:45866
I20260926 15:09:21.732199 12427 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 17156
I20260926 15:09:21.743080 17585 tablet_replica.cc:330] T bb3de9c027d947189e1264533feb1018 P 7a7a765a539f4c83adb6dcf22e590fc9: stopping tablet replica
I20260926 15:09:21.743577 17585 raft_consensus.cc:2229] T bb3de9c027d947189e1264533feb1018 P 7a7a765a539f4c83adb6dcf22e590fc9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 15:09:21.743824 17586 tablet_replica.cc:330] T bb3de9c027d947189e1264533feb1018 P 0a2b041ea3de444589f1c286ada6d129: stopping tablet replica
I20260926 15:09:21.744236 17586 raft_consensus.cc:2229] T bb3de9c027d947189e1264533feb1018 P 0a2b041ea3de444589f1c286ada6d129 [term 1 LEADER]: Raft consensus shutting down.
I20260926 15:09:21.744323 17585 raft_consensus.cc:2258] T bb3de9c027d947189e1264533feb1018 P 7a7a765a539f4c83adb6dcf22e590fc9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:09:21.745332 17586 raft_consensus.cc:2258] T bb3de9c027d947189e1264533feb1018 P 0a2b041ea3de444589f1c286ada6d129 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:09:21.745687 17584 tablet_replica.cc:330] T bb3de9c027d947189e1264533feb1018 P 22d9c22620f749a2b9db3765109f2169: stopping tablet replica
I20260926 15:09:21.746069 17584 raft_consensus.cc:2229] T bb3de9c027d947189e1264533feb1018 P 22d9c22620f749a2b9db3765109f2169 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 15:09:21.746549 17584 raft_consensus.cc:2258] T bb3de9c027d947189e1264533feb1018 P 22d9c22620f749a2b9db3765109f2169 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:09:21.747320 17585 ts_tablet_manager.cc:1905] T bb3de9c027d947189e1264533feb1018 P 7a7a765a539f4c83adb6dcf22e590fc9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:09:21.747540 17560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:21.733853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45062 (local address 127.12.34.193:33093)
0926 15:09:21.734072 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:21.734081 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:21.745032 (+ 10951us) server_negotiation.cc:407] Connection header received
0926 15:09:21.745253 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:21.745260 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:21.745393 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:21.745525 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:09:21.746722 (+ 1197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:09:21.746728 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:09:21.746733 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:09:21.746754 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:09:21.746792 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:09:21.746808 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:09:21.746910 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 15:09:21.747151 (+ 241us) server_negotiation.cc:299] Negotiation successful
0926 15:09:21.747216 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 15:09:21.748775 17278 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:09:21.753806 17586 ts_tablet_manager.cc:1905] T bb3de9c027d947189e1264533feb1018 P 0a2b041ea3de444589f1c286ada6d129: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:09:21.756161 17585 ts_tablet_manager.cc:1918] T bb3de9c027d947189e1264533feb1018 P 7a7a765a539f4c83adb6dcf22e590fc9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 15:09:21.756327 17585 log.cc:1196] T bb3de9c027d947189e1264533feb1018 P 7a7a765a539f4c83adb6dcf22e590fc9: Deleting WAL directory at /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790435318278681-12427-0/minicluster-data/ts-1/wal/wals/bb3de9c027d947189e1264533feb1018
I20260926 15:09:21.756901 17585 ts_tablet_manager.cc:1939] T bb3de9c027d947189e1264533feb1018 P 7a7a765a539f4c83adb6dcf22e590fc9: Deleting consensus metadata
I20260926 15:09:21.758889 17083 catalog_manager.cc:4841] TS 7a7a765a539f4c83adb6dcf22e590fc9 (127.12.34.194:39227): tablet bb3de9c027d947189e1264533feb1018 (table test-table [id=6efca336ccd04fdfbded71d8f467fec7]) successfully deleted
I20260926 15:09:21.760907 17586 ts_tablet_manager.cc:1918] T bb3de9c027d947189e1264533feb1018 P 0a2b041ea3de444589f1c286ada6d129: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 15:09:21.761142 17586 log.cc:1196] T bb3de9c027d947189e1264533feb1018 P 0a2b041ea3de444589f1c286ada6d129: Deleting WAL directory at /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790435318278681-12427-0/minicluster-data/ts-2/wal/wals/bb3de9c027d947189e1264533feb1018
I20260926 15:09:21.761667 17586 ts_tablet_manager.cc:1939] T bb3de9c027d947189e1264533feb1018 P 0a2b041ea3de444589f1c286ada6d129: Deleting consensus metadata
I20260926 15:09:21.763234 17084 catalog_manager.cc:4841] TS 0a2b041ea3de444589f1c286ada6d129 (127.12.34.195:38283): tablet bb3de9c027d947189e1264533feb1018 (table test-table [id=6efca336ccd04fdfbded71d8f467fec7]) successfully deleted
I20260926 15:09:21.748767 17584 ts_tablet_manager.cc:1905] T bb3de9c027d947189e1264533feb1018 P 22d9c22620f749a2b9db3765109f2169: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:09:21.858394 12427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8uoJGv/build/asan/bin/kudu with pid 17156
I20260926 15:09:21.875701 12427 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 17287
W20260926 15:09:21.875777 17082 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.34.193:33093 (error 108)
I20260926 15:09:21.877935 17559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:21.877477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.193:33093 (local address 127.0.0.1:45078)
0926 15:09:21.877643 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0926 15:09:21.877767 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.193:33093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260926 15:09:21.878147 17082 catalog_manager.cc:4584] TS 22d9c22620f749a2b9db3765109f2169 (127.12.34.193:33093): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bb3de9c027d947189e1264533feb1018: Network error: Client connection negotiation failed: client connection to 127.12.34.193:33093: connect: Connection refused (error 111)
I20260926 15:09:21.878916 17557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:21.876706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57158 (local address 127.12.34.194:39227)
0926 15:09:21.876884 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:21.876891 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:21.876914 (+ 23us) server_negotiation.cc:407] Connection header received
0926 15:09:21.877133 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:21.877141 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:21.877252 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:21.877372 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:09:21.878323 (+ 951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:09:21.878330 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:09:21.878335 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:09:21.878353 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:09:21.878388 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:09:21.878403 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:09:21.878529 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0926 15:09:21.878590 (+ 61us) server_negotiation.cc:299] Negotiation successful
0926 15:09:21.878644 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 15:09:21.879629 17409 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:09:21.931898 17559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:21.931390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.193:33093 (local address 127.0.0.1:45082)
0926 15:09:21.931610 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0926 15:09:21.931712 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.193:33093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260926 15:09:21.932981 17559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:21.932593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.193:33093 (local address 127.0.0.1:45092)
0926 15:09:21.932746 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0926 15:09:21.932848 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.193:33093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260926 15:09:21.976501 12427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8uoJGv/build/asan/bin/kudu with pid 17287
I20260926 15:09:21.993709 17559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:21.993146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.193:33093 (local address 127.0.0.1:45096)
0926 15:09:21.993394 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0926 15:09:21.993525 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.193:33093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260926 15:09:21.994347 12427 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 17417
I20260926 15:09:21.995050 17559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:21.994655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.193:33093 (local address 127.0.0.1:45100)
0926 15:09:21.994824 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0926 15:09:21.994905 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.193:33093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260926 15:09:21.997486 17561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:21.995412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45886 (local address 127.12.34.195:38283)
0926 15:09:21.995612 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:21.995618 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:21.995638 (+ 20us) server_negotiation.cc:407] Connection header received
0926 15:09:21.995723 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:21.995729 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:21.995789 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:21.995906 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:09:21.996815 (+ 909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:09:21.996821 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:09:21.996827 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:09:21.996853 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:09:21.996889 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:09:21.996904 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:09:21.997061 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0926 15:09:21.997133 (+ 72us) server_negotiation.cc:299] Negotiation successful
0926 15:09:21.997195 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 15:09:21.998198 17539 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:09:22.076027 17559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:22.075425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.193:33093 (local address 127.0.0.1:45106)
0926 15:09:22.075731 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0926 15:09:22.075830 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.193:33093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177}
I20260926 15:09:22.077232 17559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:22.076801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.193:33093 (local address 127.0.0.1:45112)
0926 15:09:22.076996 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0926 15:09:22.077109 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.193:33093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260926 15:09:22.118566 12427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8uoJGv/build/asan/bin/kudu with pid 17417
I20260926 15:09:22.137723 12427 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 17066
I20260926 15:09:22.138830 17127 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:09:22.242719 12427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8uoJGv/build/asan/bin/kudu with pid 17066
2026-09-26T15:09:22Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2764 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260926 15:09:22.283849 12427 mini_oidc.cc:152] Starting JWKS server
I20260926 15:09:22.286586 12427 webserver.cc:460] Webserver started at https://127.0.0.1:35891/ using document root <none> and password file <none>
I20260926 15:09:22.286932 12427 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 15:09:22.287632 12427 webserver.cc:460] Webserver started at http://127.0.0.1:46679/ using document root <none> and password file <none>
2026-09-26T15:09:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T15:09:22Z Disabled control of system clock
I20260926 15:09:22.314531 12427 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8uoJGv/build/asan/bin/kudu
/tmp/dist-test-task8uoJGv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790435318278681-12427-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790435318278681-12427-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790435318278681-12427-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.34.254:33963
--webserver_interface=127.12.34.254
--webserver_port=0
--builtin_ntp_servers=127.12.34.212:41651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.34.254:33963
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35891/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790435318278681-12427-0/testchainca.cert with env {}
W20260926 15:09:22.635246 17598 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:09:22.635504 17598 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:09:22.635586 17598 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:09:22.642472 17598 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:09:22.642557 17598 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:09:22.642593 17598 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:09:22.642647 17598 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:09:22.642691 17598 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 15:09:22.642742 17598 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 15:09:22.642781 17598 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:35891/jwks.json
I20260926 15:09:22.652333 17598 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.34.212:41651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790435318278681-12427-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790435318278681-12427-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.34.254:33963
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.34.254:33963
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35891/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.34.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790435318278681-12427-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790435318278681-12427-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:09:22.653448 17598 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:09:22.654568 17598 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:09:22.665803 17603 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:09:22.665921 17604 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:09:22.666273 17606 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:09:22.666769 17598 server_base.cc:1034] running on GCE node
I20260926 15:09:22.667495 17598 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:09:22.668772 17598 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:09:22.670006 17598 hybrid_clock.cc:648] HybridClock initialized: now 1790435362669953 us; error 39 us; skew 500 ppm
I20260926 15:09:22.670367 17598 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 15:09:22.672737 17598 webserver.cc:460] Webserver started at http://127.12.34.254:34289/ using document root <none> and password file <none>
I20260926 15:09:22.673451 17598 fs_manager.cc:359] Metadata directory not provided
I20260926 15:09:22.673578 17598 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:09:22.673866 17598 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:09:22.677023 17598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790435318278681-12427-0/minicluster-data/master-0/data/instance:
uuid: "50751767b7f4406db14aeb05a23ef23a"
format_stamp: "Formatted at 2026-09-26 15:09:22 on dist-test-slave-865v"
server_key: "696c871aa1b591ba20f80f7a3519f0f0"
server_key_iv: "d9060a5d08739bb1b15e40da6e7c49ec"
server_key_version: "encryptionkey@0"
I20260926 15:09:22.677834 17598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790435318278681-12427-0/minicluster-data/master-0/wal/instance:
uuid: "50751767b7f4406db14aeb05a23ef23a"
format_stamp: "Formatted at 2026-09-26 15:09:22 on dist-test-slave-865v"
server_key: "696c871aa1b591ba20f80f7a3519f0f0"
server_key_iv: "d9060a5d08739bb1b15e40da6e7c49ec"
server_key_version: "encryptionkey@0"
I20260926 15:09:22.682502 17598 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 15:09:22.685667 17612 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:09:22.686790 17598 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 15:09:22.686978 17598 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790435318278681-12427-0/minicluster-data/master-0/data,/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790435318278681-12427-0/minicluster-data/master-0/wal
uuid: "50751767b7f4406db14aeb05a23ef23a"
format_stamp: "Formatted at 2026-09-26 15:09:22 on dist-test-slave-865v"
server_key: "696c871aa1b591ba20f80f7a3519f0f0"
server_key_iv: "d9060a5d08739bb1b15e40da6e7c49ec"
server_key_version: "encryptionkey@0"
I20260926 15:09:22.687173 17598 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790435318278681-12427-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790435318278681-12427-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790435318278681-12427-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:09:22.704715 17598 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:09:22.705570 17598 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:09:22.705863 17598 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:09:22.723547 17598 rpc_server.cc:307] RPC server started. Bound to: 127.12.34.254:33963
I20260926 15:09:22.723565 17663 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.34.254:33963 every 8 connection(s)
I20260926 15:09:22.724793 17598 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
I20260926 15:09:22.728220 17664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:09:22.731112 12427 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8uoJGv/build/asan/bin/kudu as pid 17598
I20260926 15:09:22.731343 12427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790435318278681-12427-0/minicluster-data/master-0/wal/instance
I20260926 15:09:22.732090 12427 external_mini_cluster.cc:1428] Setting key 4346ad308b9fbb900ad225501f33dada
I20260926 15:09:22.740187 17664 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50751767b7f4406db14aeb05a23ef23a: Bootstrap starting.
I20260926 15:09:22.743640 17664 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 50751767b7f4406db14aeb05a23ef23a: Neither blocks nor log segments found. Creating new log.
I20260926 15:09:22.744678 17664 log.cc:824] T 00000000000000000000000000000000 P 50751767b7f4406db14aeb05a23ef23a: Log is configured to *not* fsync() on all Append() calls
I20260926 15:09:22.745172 17667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:22.733898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39658 (local address 127.12.34.254:33963)
0926 15:09:22.734782 (+ 884us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:22.734793 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:22.734829 (+ 36us) server_negotiation.cc:407] Connection header received
0926 15:09:22.736397 (+ 1568us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:22.736418 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:22.736832 (+ 414us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:22.737204 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:09:22.737759 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:22.739006 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:22.740810 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:22.741498 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:22.741630 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:22.742866 (+ 1236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:09:22.742874 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:09:22.742886 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:09:22.743379 (+ 493us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:09:22.743470 (+ 91us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:09:22.743489 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:09:22.743573 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 15:09:22.743858 (+ 285us) server_negotiation.cc:299] Negotiation successful
0926 15:09:22.744174 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":520,"thread_start_us":332,"threads_started":1}
I20260926 15:09:22.747660 17664 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50751767b7f4406db14aeb05a23ef23a: No bootstrap required, opened a new log
I20260926 15:09:22.748912 12427 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 15:09:22.755350 17664 raft_consensus.cc:348] T 00000000000000000000000000000000 P 50751767b7f4406db14aeb05a23ef23a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50751767b7f4406db14aeb05a23ef23a" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 33963 } }
I20260926 15:09:22.755694 17664 raft_consensus.cc:374] T 00000000000000000000000000000000 P 50751767b7f4406db14aeb05a23ef23a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:09:22.755795 17664 raft_consensus.cc:729] T 00000000000000000000000000000000 P 50751767b7f4406db14aeb05a23ef23a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50751767b7f4406db14aeb05a23ef23a, State: Initialized, Role: FOLLOWER
I20260926 15:09:22.756294 17664 consensus_queue.cc:260] T 00000000000000000000000000000000 P 50751767b7f4406db14aeb05a23ef23a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50751767b7f4406db14aeb05a23ef23a" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 33963 } }
I20260926 15:09:22.756551 17664 raft_consensus.cc:388] T 00000000000000000000000000000000 P 50751767b7f4406db14aeb05a23ef23a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 15:09:22.756691 17664 raft_consensus.cc:482] T 00000000000000000000000000000000 P 50751767b7f4406db14aeb05a23ef23a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 15:09:22.756855 17664 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 50751767b7f4406db14aeb05a23ef23a [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:09:22.759372 17664 raft_consensus.cc:504] T 00000000000000000000000000000000 P 50751767b7f4406db14aeb05a23ef23a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50751767b7f4406db14aeb05a23ef23a" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 33963 } }
I20260926 15:09:22.759845 17664 leader_election.cc:302] T 00000000000000000000000000000000 P 50751767b7f4406db14aeb05a23ef23a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 50751767b7f4406db14aeb05a23ef23a; no voters:
I20260926 15:09:22.760412 17664 leader_election.cc:288] T 00000000000000000000000000000000 P 50751767b7f4406db14aeb05a23ef23a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 15:09:22.760596 17669 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 50751767b7f4406db14aeb05a23ef23a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:09:22.761358 17669 raft_consensus.cc:686] T 00000000000000000000000000000000 P 50751767b7f4406db14aeb05a23ef23a [term 1 LEADER]: Becoming Leader. State: Replica: 50751767b7f4406db14aeb05a23ef23a, State: Running, Role: LEADER
I20260926 15:09:22.761905 17669 consensus_queue.cc:237] T 00000000000000000000000000000000 P 50751767b7f4406db14aeb05a23ef23a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50751767b7f4406db14aeb05a23ef23a" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 33963 } }
I20260926 15:09:22.762395 17664 sys_catalog.cc:564] T 00000000000000000000000000000000 P 50751767b7f4406db14aeb05a23ef23a [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:09:22.764582 17670 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50751767b7f4406db14aeb05a23ef23a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "50751767b7f4406db14aeb05a23ef23a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50751767b7f4406db14aeb05a23ef23a" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 33963 } } }
I20260926 15:09:22.764679 17671 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50751767b7f4406db14aeb05a23ef23a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 50751767b7f4406db14aeb05a23ef23a. Latest consensus state: current_term: 1 leader_uuid: "50751767b7f4406db14aeb05a23ef23a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50751767b7f4406db14aeb05a23ef23a" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 33963 } } }
I20260926 15:09:22.764947 17670 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50751767b7f4406db14aeb05a23ef23a [sys.catalog]: This master's current role is: LEADER
I20260926 15:09:22.765829 17671 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50751767b7f4406db14aeb05a23ef23a [sys.catalog]: This master's current role is: LEADER
I20260926 15:09:22.768559 17674 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:09:22.772888 17674 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 15:09:22.780474 17674 catalog_manager.cc:1347] Generated new cluster ID: c2b9a51d5dd94be484d7493f86af9bb1
I20260926 15:09:22.780676 17674 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 15:09:22.820148 17674 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 15:09:22.821936 17674 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:09:22.833423 17674 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 50751767b7f4406db14aeb05a23ef23a: Generated new TSK 0
I20260926 15:09:22.834353 17674 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260926 15:09:22.875790 17694 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260926 15:09:22.875968 17694 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260926 15:09:22.876895 17667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:22.855388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39670 (local address 127.12.34.254:33963)
0926 15:09:22.855647 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:22.855654 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:22.856117 (+ 463us) server_negotiation.cc:407] Connection header received
0926 15:09:22.856470 (+ 353us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:22.856474 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:22.856544 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:22.856666 (+ 122us) server_negotiation.cc:226] Negotiated authn=JWT
0926 15:09:22.857326 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:22.858108 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:22.860188 (+ 2080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:22.860791 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:22.860897 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:22.861579 (+ 682us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 15:09:22.876427 (+ 14848us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35891/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0926 15:09:22.876677 (+ 250us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:39670: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35891/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":73}
W20260926 15:09:22.877048 17667 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:39670: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35891/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260926 15:09:22.879032 17693 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:09:22.855297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:33963 (local address 127.0.0.1:39670)
0926 15:09:22.856000 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0926 15:09:22.856021 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 15:09:22.856204 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:09:22.856720 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:09:22.856725 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:09:22.856738 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0926 15:09:22.857146 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:22.857158 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:22.858254 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:22.858260 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:22.860034 (+ 1774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:22.860048 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:22.861041 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:22.861048 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:22.861260 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:22.861332 (+ 72us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 15:09:22.876682 (+ 15350us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35891/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0926 15:09:22.877027 (+ 345us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.12.34.254:33963: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35891/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":353,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.12.34.254:33963: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35891/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260926 15:09:22.881188 12427 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 17598
I20260926 15:09:22.882215 17659 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:09:22.976444 12427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8uoJGv/build/asan/bin/kudu with pid 17598
2026-09-26T15:09:23Z chronyd exiting
I20260926 15:09:23.027058 12427 test_util.cc:175] -----------------------------------------------
I20260926 15:09:23.027209 12427 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790435318278681-12427-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (747 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260926 15:09:23.030783 12427 mini_oidc.cc:152] Starting JWKS server
I20260926 15:09:23.032485 12427 webserver.cc:460] Webserver started at https://127.0.0.1:35393/ using document root <none> and password file <none>
I20260926 15:09:23.032648 12427 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 15:09:23.033325 12427 webserver.cc:460] Webserver started at http://127.0.0.1:36085/ using document root <none> and password file <none>
2026-09-26T15:09:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T15:09:23Z Disabled control of system clock
I20260926 15:09:23.058743 12427 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8uoJGv/build/asan/bin/kudu
/tmp/dist-test-task8uoJGv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790435318278681-12427-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790435318278681-12427-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790435318278681-12427-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.34.254:41379
--webserver_interface=127.12.34.254
--webserver_port=0
--builtin_ntp_servers=127.12.34.212:43515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.34.254:41379
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35393/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790435318278681-12427-0/testchainca.cert with env {}
W20260926 15:09:23.373207 17701 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:09:23.373461 17701 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:09:23.373544 17701 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:09:23.380467 17701 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:09:23.380556 17701 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:09:23.380594 17701 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:09:23.380641 17701 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:09:23.380686 17701 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 15:09:23.380739 17701 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 15:09:23.380779 17701 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:35393/jwks.json
I20260926 15:09:23.390122 17701 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.34.212:43515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790435318278681-12427-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790435318278681-12427-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.34.254:41379
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.34.254:41379
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35393/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.34.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790435318278681-12427-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790435318278681-12427-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:09:23.391227 17701 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:09:23.392354 17701 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:09:23.404234 17709 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:09:23.404321 17707 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:09:23.404496 17706 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:09:23.410465 17701 server_base.cc:1034] running on GCE node
I20260926 15:09:23.411301 17701 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:09:23.412707 17701 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:09:23.413972 17701 hybrid_clock.cc:648] HybridClock initialized: now 1790435363413936 us; error 51 us; skew 500 ppm
I20260926 15:09:23.414369 17701 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 15:09:23.416586 17701 webserver.cc:460] Webserver started at http://127.12.34.254:42121/ using document root <none> and password file <none>
I20260926 15:09:23.417358 17701 fs_manager.cc:359] Metadata directory not provided
I20260926 15:09:23.417483 17701 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:09:23.417819 17701 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:09:23.420531 17701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790435318278681-12427-0/minicluster-data/master-0/data/instance:
uuid: "c0794d27c3734eb09d77910971afa9ef"
format_stamp: "Formatted at 2026-09-26 15:09:23 on dist-test-slave-865v"
server_key: "9cf6997db96ff5b18c1789dc790054c3"
server_key_iv: "6d0392fc9d3bebfe9f5a08ee9ddf882f"
server_key_version: "encryptionkey@0"
I20260926 15:09:23.421373 17701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790435318278681-12427-0/minicluster-data/master-0/wal/instance:
uuid: "c0794d27c3734eb09d77910971afa9ef"
format_stamp: "Formatted at 2026-09-26 15:09:23 on dist-test-slave-865v"
server_key: "9cf6997db96ff5b18c1789dc790054c3"
server_key_iv: "6d0392fc9d3bebfe9f5a08ee9ddf882f"
server_key_version: "encryptionkey@0"
I20260926 15:09:23.426224 17701 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 15:09:23.429980 17715 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:09:23.431363 17701 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260926 15:09:23.431579 17701 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790435318278681-12427-0/minicluster-data/master-0/data,/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790435318278681-12427-0/minicluster-data/master-0/wal
uuid: "c0794d27c3734eb09d77910971afa9ef"
format_stamp: "Formatted at 2026-09-26 15:09:23 on dist-test-slave-865v"
server_key: "9cf6997db96ff5b18c1789dc790054c3"
server_key_iv: "6d0392fc9d3bebfe9f5a08ee9ddf882f"
server_key_version: "encryptionkey@0"
I20260926 15:09:23.431813 17701 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790435318278681-12427-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790435318278681-12427-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790435318278681-12427-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:09:23.456705 17701 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:09:23.457612 17701 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:09:23.457943 17701 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:09:23.476876 17701 rpc_server.cc:307] RPC server started. Bound to: 127.12.34.254:41379
I20260926 15:09:23.476900 17766 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.34.254:41379 every 8 connection(s)
I20260926 15:09:23.478219 17701 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
I20260926 15:09:23.481817 17767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:09:23.485852 12427 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8uoJGv/build/asan/bin/kudu as pid 17701
I20260926 15:09:23.486042 12427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790435318278681-12427-0/minicluster-data/master-0/wal/instance
I20260926 15:09:23.486661 12427 external_mini_cluster.cc:1428] Setting key b6dcb3579345df9ba63da3f6532a7ee9
I20260926 15:09:23.493029 17767 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0794d27c3734eb09d77910971afa9ef: Bootstrap starting.
I20260926 15:09:23.496443 17767 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c0794d27c3734eb09d77910971afa9ef: Neither blocks nor log segments found. Creating new log.
I20260926 15:09:23.497696 17767 log.cc:824] T 00000000000000000000000000000000 P c0794d27c3734eb09d77910971afa9ef: Log is configured to *not* fsync() on all Append() calls
I20260926 15:09:23.500285 17767 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0794d27c3734eb09d77910971afa9ef: No bootstrap required, opened a new log
I20260926 15:09:23.507042 17767 raft_consensus.cc:348] T 00000000000000000000000000000000 P c0794d27c3734eb09d77910971afa9ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0794d27c3734eb09d77910971afa9ef" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 41379 } }
I20260926 15:09:23.507368 17767 raft_consensus.cc:374] T 00000000000000000000000000000000 P c0794d27c3734eb09d77910971afa9ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:09:23.507522 17767 raft_consensus.cc:729] T 00000000000000000000000000000000 P c0794d27c3734eb09d77910971afa9ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0794d27c3734eb09d77910971afa9ef, State: Initialized, Role: FOLLOWER
I20260926 15:09:23.508059 17767 consensus_queue.cc:260] T 00000000000000000000000000000000 P c0794d27c3734eb09d77910971afa9ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0794d27c3734eb09d77910971afa9ef" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 41379 } }
I20260926 15:09:23.508270 17767 raft_consensus.cc:388] T 00000000000000000000000000000000 P c0794d27c3734eb09d77910971afa9ef [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 15:09:23.508417 17767 raft_consensus.cc:482] T 00000000000000000000000000000000 P c0794d27c3734eb09d77910971afa9ef [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 15:09:23.508584 17767 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c0794d27c3734eb09d77910971afa9ef [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:09:23.510753 17767 raft_consensus.cc:504] T 00000000000000000000000000000000 P c0794d27c3734eb09d77910971afa9ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0794d27c3734eb09d77910971afa9ef" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 41379 } }
I20260926 15:09:23.511207 17767 leader_election.cc:302] T 00000000000000000000000000000000 P c0794d27c3734eb09d77910971afa9ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c0794d27c3734eb09d77910971afa9ef; no voters:
I20260926 15:09:23.511785 17767 leader_election.cc:288] T 00000000000000000000000000000000 P c0794d27c3734eb09d77910971afa9ef [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 15:09:23.511929 17772 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c0794d27c3734eb09d77910971afa9ef [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:09:23.512759 17772 raft_consensus.cc:686] T 00000000000000000000000000000000 P c0794d27c3734eb09d77910971afa9ef [term 1 LEADER]: Becoming Leader. State: Replica: c0794d27c3734eb09d77910971afa9ef, State: Running, Role: LEADER
I20260926 15:09:23.513295 17772 consensus_queue.cc:237] T 00000000000000000000000000000000 P c0794d27c3734eb09d77910971afa9ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0794d27c3734eb09d77910971afa9ef" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 41379 } }
I20260926 15:09:23.514055 17767 sys_catalog.cc:564] T 00000000000000000000000000000000 P c0794d27c3734eb09d77910971afa9ef [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:09:23.516454 17774 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0794d27c3734eb09d77910971afa9ef [sys.catalog]: SysCatalogTable state changed. Reason: New leader c0794d27c3734eb09d77910971afa9ef. Latest consensus state: current_term: 1 leader_uuid: "c0794d27c3734eb09d77910971afa9ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0794d27c3734eb09d77910971afa9ef" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 41379 } } }
I20260926 15:09:23.516816 17774 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0794d27c3734eb09d77910971afa9ef [sys.catalog]: This master's current role is: LEADER
I20260926 15:09:23.518172 17770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:23.488596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38078 (local address 127.12.34.254:41379)
0926 15:09:23.489620 (+ 1024us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:23.489631 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:23.489675 (+ 44us) server_negotiation.cc:407] Connection header received
0926 15:09:23.491535 (+ 1860us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:23.491557 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:23.491993 (+ 436us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:23.492529 (+ 536us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:09:23.492935 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:23.494598 (+ 1663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:23.496362 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:23.497144 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:23.497294 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:23.514146 (+ 16852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:09:23.514157 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:09:23.514182 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:09:23.514890 (+ 708us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:09:23.515112 (+ 222us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:09:23.515144 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:09:23.515259 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0926 15:09:23.516141 (+ 882us) server_negotiation.cc:299] Negotiation successful
0926 15:09:23.516571 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":654,"thread_start_us":422,"threads_started":1}
I20260926 15:09:23.521754 17773 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0794d27c3734eb09d77910971afa9ef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c0794d27c3734eb09d77910971afa9ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0794d27c3734eb09d77910971afa9ef" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 41379 } } }
I20260926 15:09:23.522083 17773 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0794d27c3734eb09d77910971afa9ef [sys.catalog]: This master's current role is: LEADER
I20260926 15:09:23.523279 17777 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:09:23.523468 12427 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 15:09:23.528429 17777 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 15:09:23.537478 17777 catalog_manager.cc:1347] Generated new cluster ID: 97a46cc4893a4bc194c898e3d048abbf
I20260926 15:09:23.537580 17777 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 15:09:23.547508 17777 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 15:09:23.549304 17777 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:09:23.565163 17777 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c0794d27c3734eb09d77910971afa9ef: Generated new TSK 0
I20260926 15:09:23.566303 17777 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260926 15:09:23.603513 17797 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260926 15:09:23.603813 17797 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260926 15:09:23.604846 17770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:23.582694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38082 (local address 127.12.34.254:41379)
0926 15:09:23.582897 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:23.582909 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:23.583256 (+ 347us) server_negotiation.cc:407] Connection header received
0926 15:09:23.583442 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:23.583448 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:23.583535 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:23.583714 (+ 179us) server_negotiation.cc:226] Negotiated authn=JWT
0926 15:09:23.584133 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:23.585262 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:23.587292 (+ 2030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:23.587995 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:23.588128 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:23.588656 (+ 528us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 15:09:23.604155 (+ 15499us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35393/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0926 15:09:23.604537 (+ 382us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:38082: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35393/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":81}
W20260926 15:09:23.604986 17770 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:38082: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35393/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260926 15:09:23.605500 17796 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:09:23.582519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:41379 (local address 127.0.0.1:38082)
0926 15:09:23.583148 (+ 629us) negotiation.cc:107] Waiting for socket to connect
0926 15:09:23.583167 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 15:09:23.583282 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:09:23.583725 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:09:23.583728 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:09:23.583737 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0926 15:09:23.583993 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:23.584000 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:23.585492 (+ 1492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:23.585498 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:23.587119 (+ 1621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:23.587132 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:23.588175 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:23.588180 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:23.588331 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:23.588364 (+ 33us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 15:09:23.604479 (+ 16115us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35393/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0926 15:09:23.604811 (+ 332us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.12.34.254:41379: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35393/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":379,"threads_started":1}
I20260926 15:09:23.607282 12427 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 17701
I20260926 15:09:23.608542 17762 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:09:23.700837 12427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8uoJGv/build/asan/bin/kudu with pid 17701
2026-09-26T15:09:23Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (724 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260926 15:09:23.755664 12427 mini_oidc.cc:152] Starting JWKS server
I20260926 15:09:23.757529 12427 webserver.cc:460] Webserver started at https://127.0.0.1:43791/ using document root <none> and password file <none>
I20260926 15:09:23.757712 12427 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 15:09:23.758282 12427 webserver.cc:460] Webserver started at http://127.0.0.1:37289/ using document root <none> and password file <none>
2026-09-26T15:09:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T15:09:23Z Disabled control of system clock
I20260926 15:09:23.783627 12427 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8uoJGv/build/asan/bin/kudu
/tmp/dist-test-task8uoJGv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790435318278681-12427-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790435318278681-12427-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790435318278681-12427-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.34.254:41253
--webserver_interface=127.12.34.254
--webserver_port=0
--builtin_ntp_servers=127.12.34.212:34435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.34.254:41253
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43791/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260926 15:09:24.094264 17804 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:09:24.094542 17804 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:09:24.094726 17804 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:09:24.103117 17804 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:09:24.103219 17804 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:09:24.103312 17804 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:09:24.103413 17804 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:09:24.103499 17804 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 15:09:24.103586 17804 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 15:09:24.103654 17804 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:43791/jwks.json
I20260926 15:09:24.112872 17804 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.34.212:34435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790435318278681-12427-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790435318278681-12427-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.34.254:41253
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.34.254:41253
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43791/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.34.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790435318278681-12427-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:09:24.114140 17804 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:09:24.115871 17804 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:09:24.127592 17809 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:09:24.127662 17812 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:09:24.127640 17810 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:09:24.128239 17804 server_base.cc:1034] running on GCE node
I20260926 15:09:24.128870 17804 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:09:24.130180 17804 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:09:24.131404 17804 hybrid_clock.cc:648] HybridClock initialized: now 1790435364131331 us; error 52 us; skew 500 ppm
I20260926 15:09:24.131740 17804 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 15:09:24.133976 17804 webserver.cc:460] Webserver started at http://127.12.34.254:42069/ using document root <none> and password file <none>
I20260926 15:09:24.134650 17804 fs_manager.cc:359] Metadata directory not provided
I20260926 15:09:24.134739 17804 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:09:24.135087 17804 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:09:24.137800 17804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790435318278681-12427-0/minicluster-data/master-0/data/instance:
uuid: "52bbdcf3267943d7a1c38ffe8adfd8a0"
format_stamp: "Formatted at 2026-09-26 15:09:24 on dist-test-slave-865v"
server_key: "903b64caa7fadda93ec70cb9e1132ec1"
server_key_iv: "f5700dbad8255005cb28b647683f2710"
server_key_version: "encryptionkey@0"
I20260926 15:09:24.138547 17804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790435318278681-12427-0/minicluster-data/master-0/wal/instance:
uuid: "52bbdcf3267943d7a1c38ffe8adfd8a0"
format_stamp: "Formatted at 2026-09-26 15:09:24 on dist-test-slave-865v"
server_key: "903b64caa7fadda93ec70cb9e1132ec1"
server_key_iv: "f5700dbad8255005cb28b647683f2710"
server_key_version: "encryptionkey@0"
I20260926 15:09:24.143286 17804 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260926 15:09:24.146512 17818 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:09:24.147799 17804 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 15:09:24.147987 17804 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790435318278681-12427-0/minicluster-data/master-0/data,/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790435318278681-12427-0/minicluster-data/master-0/wal
uuid: "52bbdcf3267943d7a1c38ffe8adfd8a0"
format_stamp: "Formatted at 2026-09-26 15:09:24 on dist-test-slave-865v"
server_key: "903b64caa7fadda93ec70cb9e1132ec1"
server_key_iv: "f5700dbad8255005cb28b647683f2710"
server_key_version: "encryptionkey@0"
I20260926 15:09:24.148197 17804 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790435318278681-12427-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790435318278681-12427-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790435318278681-12427-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:09:24.161489 17804 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:09:24.162176 17804 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:09:24.162493 17804 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:09:24.180611 17804 rpc_server.cc:307] RPC server started. Bound to: 127.12.34.254:41253
I20260926 15:09:24.180634 17869 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.34.254:41253 every 8 connection(s)
I20260926 15:09:24.182006 17804 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
I20260926 15:09:24.185598 17870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:09:24.190485 12427 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8uoJGv/build/asan/bin/kudu as pid 17804
I20260926 15:09:24.190682 12427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790435318278681-12427-0/minicluster-data/master-0/wal/instance
I20260926 15:09:24.191393 12427 external_mini_cluster.cc:1428] Setting key ba114ee08dd0f78314ed2693cb3904eb
I20260926 15:09:24.196497 17870 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52bbdcf3267943d7a1c38ffe8adfd8a0: Bootstrap starting.
I20260926 15:09:24.199363 17870 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 52bbdcf3267943d7a1c38ffe8adfd8a0: Neither blocks nor log segments found. Creating new log.
I20260926 15:09:24.200330 17870 log.cc:824] T 00000000000000000000000000000000 P 52bbdcf3267943d7a1c38ffe8adfd8a0: Log is configured to *not* fsync() on all Append() calls
I20260926 15:09:24.203011 17870 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52bbdcf3267943d7a1c38ffe8adfd8a0: No bootstrap required, opened a new log
I20260926 15:09:24.203783 17873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:24.193289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46404 (local address 127.12.34.254:41253)
0926 15:09:24.194265 (+ 976us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:24.194278 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:24.194317 (+ 39us) server_negotiation.cc:407] Connection header received
0926 15:09:24.195873 (+ 1556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:24.195894 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:24.196296 (+ 402us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:24.196659 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:09:24.197199 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:24.198424 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:24.199708 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:24.200334 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:24.200536 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:24.201672 (+ 1136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:09:24.201680 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:09:24.201692 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:09:24.202180 (+ 488us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:09:24.202250 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:09:24.202269 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:09:24.202394 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0926 15:09:24.202604 (+ 210us) server_negotiation.cc:299] Negotiation successful
0926 15:09:24.202880 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":572,"thread_start_us":340,"threads_started":1}
I20260926 15:09:24.207518 12427 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 15:09:24.211426 17870 raft_consensus.cc:348] T 00000000000000000000000000000000 P 52bbdcf3267943d7a1c38ffe8adfd8a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52bbdcf3267943d7a1c38ffe8adfd8a0" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 41253 } }
I20260926 15:09:24.211767 17870 raft_consensus.cc:374] T 00000000000000000000000000000000 P 52bbdcf3267943d7a1c38ffe8adfd8a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:09:24.211894 17870 raft_consensus.cc:729] T 00000000000000000000000000000000 P 52bbdcf3267943d7a1c38ffe8adfd8a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52bbdcf3267943d7a1c38ffe8adfd8a0, State: Initialized, Role: FOLLOWER
I20260926 15:09:24.212402 17870 consensus_queue.cc:260] T 00000000000000000000000000000000 P 52bbdcf3267943d7a1c38ffe8adfd8a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52bbdcf3267943d7a1c38ffe8adfd8a0" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 41253 } }
I20260926 15:09:24.212623 17870 raft_consensus.cc:388] T 00000000000000000000000000000000 P 52bbdcf3267943d7a1c38ffe8adfd8a0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 15:09:24.212764 17870 raft_consensus.cc:482] T 00000000000000000000000000000000 P 52bbdcf3267943d7a1c38ffe8adfd8a0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 15:09:24.212944 17870 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 52bbdcf3267943d7a1c38ffe8adfd8a0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:09:24.215261 17870 raft_consensus.cc:504] T 00000000000000000000000000000000 P 52bbdcf3267943d7a1c38ffe8adfd8a0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52bbdcf3267943d7a1c38ffe8adfd8a0" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 41253 } }
I20260926 15:09:24.215667 17870 leader_election.cc:302] T 00000000000000000000000000000000 P 52bbdcf3267943d7a1c38ffe8adfd8a0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 52bbdcf3267943d7a1c38ffe8adfd8a0; no voters:
I20260926 15:09:24.216144 17870 leader_election.cc:288] T 00000000000000000000000000000000 P 52bbdcf3267943d7a1c38ffe8adfd8a0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 15:09:24.216286 17875 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 52bbdcf3267943d7a1c38ffe8adfd8a0 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:09:24.216958 17875 raft_consensus.cc:686] T 00000000000000000000000000000000 P 52bbdcf3267943d7a1c38ffe8adfd8a0 [term 1 LEADER]: Becoming Leader. State: Replica: 52bbdcf3267943d7a1c38ffe8adfd8a0, State: Running, Role: LEADER
I20260926 15:09:24.217460 17875 consensus_queue.cc:237] T 00000000000000000000000000000000 P 52bbdcf3267943d7a1c38ffe8adfd8a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52bbdcf3267943d7a1c38ffe8adfd8a0" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 41253 } }
I20260926 15:09:24.218240 17870 sys_catalog.cc:564] T 00000000000000000000000000000000 P 52bbdcf3267943d7a1c38ffe8adfd8a0 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:09:24.220676 17877 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52bbdcf3267943d7a1c38ffe8adfd8a0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 52bbdcf3267943d7a1c38ffe8adfd8a0. Latest consensus state: current_term: 1 leader_uuid: "52bbdcf3267943d7a1c38ffe8adfd8a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52bbdcf3267943d7a1c38ffe8adfd8a0" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 41253 } } }
I20260926 15:09:24.221122 17877 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52bbdcf3267943d7a1c38ffe8adfd8a0 [sys.catalog]: This master's current role is: LEADER
I20260926 15:09:24.221688 17876 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52bbdcf3267943d7a1c38ffe8adfd8a0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "52bbdcf3267943d7a1c38ffe8adfd8a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52bbdcf3267943d7a1c38ffe8adfd8a0" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 41253 } } }
I20260926 15:09:24.221988 17876 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52bbdcf3267943d7a1c38ffe8adfd8a0 [sys.catalog]: This master's current role is: LEADER
I20260926 15:09:24.222797 17884 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:09:24.227640 17884 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 15:09:24.236276 17884 catalog_manager.cc:1347] Generated new cluster ID: fbffdf09d0e542f9bd640a3d3b51b5bd
I20260926 15:09:24.236400 17884 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 15:09:24.246738 17884 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 15:09:24.247958 17884 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:09:24.259721 17884 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 52bbdcf3267943d7a1c38ffe8adfd8a0: Generated new TSK 0
I20260926 15:09:24.260525 17884 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 15:09:24.287093 17873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:24.278779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46412 (local address 127.12.34.254:41253)
0926 15:09:24.278953 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:24.278960 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:24.279329 (+ 369us) server_negotiation.cc:407] Connection header received
0926 15:09:24.279548 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:24.279554 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:24.279651 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:24.279798 (+ 147us) server_negotiation.cc:226] Negotiated authn=JWT
0926 15:09:24.280203 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:24.281336 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:24.282767 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:24.283489 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:24.283618 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:24.284098 (+ 480us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 15:09:24.286466 (+ 2368us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43791/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0926 15:09:24.286763 (+ 297us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:46412: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43791/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":66}
W20260926 15:09:24.287220 17873 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:46412: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43791/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260926 15:09:24.289150 17899 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:09:24.278670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:41253 (local address 127.0.0.1:46412)
0926 15:09:24.279234 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0926 15:09:24.279250 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 15:09:24.279358 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:09:24.279810 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:09:24.279813 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:09:24.279821 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0926 15:09:24.280068 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:24.280075 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:24.281501 (+ 1426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:24.281506 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:24.282623 (+ 1117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:24.282633 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:24.283685 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:24.283690 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:24.283813 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:24.283844 (+ 31us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 15:09:24.286683 (+ 2839us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43791/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0926 15:09:24.288729 (+ 2046us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.12.34.254:41253: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43791/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":457,"mutex_wait_us":1745,"thread_start_us":304,"threads_started":1}
I20260926 15:09:24.290114 12427 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 17804
I20260926 15:09:24.291582 17865 generic_service.cc:196] Checking for leaks (request via RPC)
E20260926 15:09:24.302693 17900 webserver.cc:533] Webserver: sslize failed:
E20260926 15:09:24.302814 17900 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260926 15:09:24.384320 12427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8uoJGv/build/asan/bin/kudu with pid 17804
2026-09-26T15:09:24Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (683 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-26T15:09:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T15:09:24Z Disabled control of system clock
I20260926 15:09:24.463907 12427 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8uoJGv/build/asan/bin/kudu
/tmp/dist-test-task8uoJGv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790435318278681-12427-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790435318278681-12427-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790435318278681-12427-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.34.254:36017
--webserver_interface=127.12.34.254
--webserver_port=0
--builtin_ntp_servers=127.12.34.212:42369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.34.254:36017
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790435318278681-12427-0/testchainca.cert with env {}
W20260926 15:09:24.779012 17905 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:09:24.779315 17905 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:09:24.779469 17905 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:09:24.786828 17905 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:09:24.786955 17905 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:09:24.787043 17905 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:09:24.787153 17905 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:09:24.787233 17905 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 15:09:24.787336 17905 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 15:09:24.787483 17905 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260926 15:09:24.798611 17905 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.34.212:42369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790435318278681-12427-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790435318278681-12427-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.34.254:36017
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.34.254:36017
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.34.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790435318278681-12427-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790435318278681-12427-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:09:24.800011 17905 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:09:24.801369 17905 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:09:24.813896 17911 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:09:24.813982 17910 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:09:24.813987 17913 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:09:24.814845 17905 server_base.cc:1034] running on GCE node
I20260926 15:09:24.815636 17905 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:09:24.817062 17905 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:09:24.818352 17905 hybrid_clock.cc:648] HybridClock initialized: now 1790435364818320 us; error 30 us; skew 500 ppm
I20260926 15:09:24.818717 17905 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 15:09:24.821063 17905 webserver.cc:460] Webserver started at http://127.12.34.254:38645/ using document root <none> and password file <none>
I20260926 15:09:24.821805 17905 fs_manager.cc:359] Metadata directory not provided
I20260926 15:09:24.821892 17905 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:09:24.822176 17905 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:09:24.824865 17905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790435318278681-12427-0/minicluster-data/master-0/data/instance:
uuid: "f522631742c54d92b6029c1eb38e279f"
format_stamp: "Formatted at 2026-09-26 15:09:24 on dist-test-slave-865v"
server_key: "c0fc6d69d93f50a1fe457e85df1daad3"
server_key_iv: "d5b750140f8036b5f12741e526e06edc"
server_key_version: "encryptionkey@0"
I20260926 15:09:24.825717 17905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790435318278681-12427-0/minicluster-data/master-0/wal/instance:
uuid: "f522631742c54d92b6029c1eb38e279f"
format_stamp: "Formatted at 2026-09-26 15:09:24 on dist-test-slave-865v"
server_key: "c0fc6d69d93f50a1fe457e85df1daad3"
server_key_iv: "d5b750140f8036b5f12741e526e06edc"
server_key_version: "encryptionkey@0"
I20260926 15:09:24.831131 17905 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 15:09:24.834699 17919 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:09:24.836097 17905 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260926 15:09:24.836310 17905 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790435318278681-12427-0/minicluster-data/master-0/data,/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790435318278681-12427-0/minicluster-data/master-0/wal
uuid: "f522631742c54d92b6029c1eb38e279f"
format_stamp: "Formatted at 2026-09-26 15:09:24 on dist-test-slave-865v"
server_key: "c0fc6d69d93f50a1fe457e85df1daad3"
server_key_iv: "d5b750140f8036b5f12741e526e06edc"
server_key_version: "encryptionkey@0"
I20260926 15:09:24.836539 17905 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790435318278681-12427-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790435318278681-12427-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790435318278681-12427-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:09:24.869169 17905 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:09:24.870012 17905 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:09:24.870328 17905 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:09:24.888373 17905 rpc_server.cc:307] RPC server started. Bound to: 127.12.34.254:36017
I20260926 15:09:24.888399 17970 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.34.254:36017 every 8 connection(s)
I20260926 15:09:24.889639 17905 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
I20260926 15:09:24.891028 12427 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8uoJGv/build/asan/bin/kudu as pid 17905
I20260926 15:09:24.891206 12427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790435318278681-12427-0/minicluster-data/master-0/wal/instance
I20260926 15:09:24.891849 12427 external_mini_cluster.cc:1428] Setting key ead64743f3157a8bd46f54aff53780f9
I20260926 15:09:24.893821 17971 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:09:24.906769 17974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:24.893731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34274 (local address 127.12.34.254:36017)
0926 15:09:24.894767 (+ 1036us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:24.894783 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:24.894834 (+ 51us) server_negotiation.cc:407] Connection header received
0926 15:09:24.896829 (+ 1995us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:24.896854 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:24.897439 (+ 585us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:24.897856 (+ 417us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:09:24.898351 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:24.900412 (+ 2061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:24.901778 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:24.902773 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:24.902949 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:24.904032 (+ 1083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:09:24.904043 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:09:24.904065 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:09:24.904775 (+ 710us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:09:24.904861 (+ 86us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:09:24.904885 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:09:24.905005 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0926 15:09:24.905267 (+ 262us) server_negotiation.cc:299] Negotiation successful
0926 15:09:24.905575 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":551,"thread_start_us":352,"threads_started":1}
I20260926 15:09:24.907958 17971 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f522631742c54d92b6029c1eb38e279f: Bootstrap starting.
I20260926 15:09:24.910364 12427 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 15:09:24.911188 17971 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f522631742c54d92b6029c1eb38e279f: Neither blocks nor log segments found. Creating new log.
I20260926 15:09:24.912333 17971 log.cc:824] T 00000000000000000000000000000000 P f522631742c54d92b6029c1eb38e279f: Log is configured to *not* fsync() on all Append() calls
I20260926 15:09:24.915280 17971 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f522631742c54d92b6029c1eb38e279f: No bootstrap required, opened a new log
I20260926 15:09:24.922317 17971 raft_consensus.cc:348] T 00000000000000000000000000000000 P f522631742c54d92b6029c1eb38e279f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f522631742c54d92b6029c1eb38e279f" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 36017 } }
I20260926 15:09:24.922619 17971 raft_consensus.cc:374] T 00000000000000000000000000000000 P f522631742c54d92b6029c1eb38e279f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:09:24.922689 17971 raft_consensus.cc:729] T 00000000000000000000000000000000 P f522631742c54d92b6029c1eb38e279f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f522631742c54d92b6029c1eb38e279f, State: Initialized, Role: FOLLOWER
I20260926 15:09:24.923132 17971 consensus_queue.cc:260] T 00000000000000000000000000000000 P f522631742c54d92b6029c1eb38e279f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f522631742c54d92b6029c1eb38e279f" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 36017 } }
I20260926 15:09:24.923317 17971 raft_consensus.cc:388] T 00000000000000000000000000000000 P f522631742c54d92b6029c1eb38e279f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 15:09:24.923411 17971 raft_consensus.cc:482] T 00000000000000000000000000000000 P f522631742c54d92b6029c1eb38e279f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 15:09:24.923532 17971 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f522631742c54d92b6029c1eb38e279f [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:09:24.925657 17971 raft_consensus.cc:504] T 00000000000000000000000000000000 P f522631742c54d92b6029c1eb38e279f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f522631742c54d92b6029c1eb38e279f" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 36017 } }
I20260926 15:09:24.926146 17971 leader_election.cc:302] T 00000000000000000000000000000000 P f522631742c54d92b6029c1eb38e279f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f522631742c54d92b6029c1eb38e279f; no voters:
I20260926 15:09:24.926666 17971 leader_election.cc:288] T 00000000000000000000000000000000 P f522631742c54d92b6029c1eb38e279f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 15:09:24.926810 17976 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f522631742c54d92b6029c1eb38e279f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:09:24.927500 17976 raft_consensus.cc:686] T 00000000000000000000000000000000 P f522631742c54d92b6029c1eb38e279f [term 1 LEADER]: Becoming Leader. State: Replica: f522631742c54d92b6029c1eb38e279f, State: Running, Role: LEADER
I20260926 15:09:24.928004 17976 consensus_queue.cc:237] T 00000000000000000000000000000000 P f522631742c54d92b6029c1eb38e279f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f522631742c54d92b6029c1eb38e279f" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 36017 } }
I20260926 15:09:24.928658 17971 sys_catalog.cc:564] T 00000000000000000000000000000000 P f522631742c54d92b6029c1eb38e279f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:09:24.930475 17977 sys_catalog.cc:455] T 00000000000000000000000000000000 P f522631742c54d92b6029c1eb38e279f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f522631742c54d92b6029c1eb38e279f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f522631742c54d92b6029c1eb38e279f" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 36017 } } }
I20260926 15:09:24.930822 17977 sys_catalog.cc:458] T 00000000000000000000000000000000 P f522631742c54d92b6029c1eb38e279f [sys.catalog]: This master's current role is: LEADER
I20260926 15:09:24.931265 17978 sys_catalog.cc:455] T 00000000000000000000000000000000 P f522631742c54d92b6029c1eb38e279f [sys.catalog]: SysCatalogTable state changed. Reason: New leader f522631742c54d92b6029c1eb38e279f. Latest consensus state: current_term: 1 leader_uuid: "f522631742c54d92b6029c1eb38e279f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f522631742c54d92b6029c1eb38e279f" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 36017 } } }
I20260926 15:09:24.931635 17978 sys_catalog.cc:458] T 00000000000000000000000000000000 P f522631742c54d92b6029c1eb38e279f [sys.catalog]: This master's current role is: LEADER
I20260926 15:09:24.933617 17981 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:09:24.938139 17981 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 15:09:24.951731 17981 catalog_manager.cc:1347] Generated new cluster ID: f45605770f7e4511809d5242901fcf5e
I20260926 15:09:24.951872 17981 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 15:09:24.971110 17981 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 15:09:24.972939 17981 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:09:24.984756 17981 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f522631742c54d92b6029c1eb38e279f: Generated new TSK 0
I20260926 15:09:24.985630 17981 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 15:09:25.075834 17974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:24.998732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34290 (local address 127.12.34.254:36017)
0926 15:09:24.998933 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:24.998940 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:24.999315 (+ 375us) server_negotiation.cc:407] Connection header received
0926 15:09:24.999585 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:24.999589 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:24.999658 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:24.999813 (+ 155us) server_negotiation.cc:226] Negotiated authn=JWT
0926 15:09:25.000242 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:25.000961 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:25.002339 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:25.002772 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:25.002916 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:25.003369 (+ 453us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 15:09:25.075052 (+ 71683us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0926 15:09:25.075505 (+ 453us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:34290: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":50}
W20260926 15:09:25.075989 17974 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:34290: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260926 15:09:25.076102 18000 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:09:24.998646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:36017 (local address 127.0.0.1:34290)
0926 15:09:24.999212 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0926 15:09:24.999228 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 15:09:24.999390 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:09:24.999824 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:09:24.999828 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:09:24.999839 (+ 11us) client_negotiation.cc:189] Negotiated authn=JWT
0926 15:09:25.000099 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:25.000107 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:25.001141 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:25.001147 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:25.002180 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:25.002190 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:25.002955 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:25.002963 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:25.003091 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:25.003121 (+ 30us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 15:09:25.075376 (+ 72255us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0926 15:09:25.075675 (+ 299us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.12.34.254:36017: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":337,"threads_started":1}
I20260926 15:09:25.077463 12427 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 17905
I20260926 15:09:25.078732 17966 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:09:25.178905 12427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8uoJGv/build/asan/bin/kudu with pid 17905
2026-09-26T15:09:25Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (774 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (400 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (405 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435318278681-12427-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 15:09:26 dist-test-slave-865v krb5kdc[18015](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 15:09:26 dist-test-slave-865v krb5kdc[18015](info): set up 2 sockets
Sep 26 15:09:26 dist-test-slave-865v krb5kdc[18015](info): commencing operation
krb5kdc: starting...
W20260926 15:09:27.089812 12427 mini_kdc.cc:121] Time spent starting KDC: real 1.068s user 0.004s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435318278681-12427-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435318278681-12427-0/krb5kdc/test-user.keytab.
Sep 26 15:09:27 dist-test-slave-865v krb5kdc[18015](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435367, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T15:09:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T15:09:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.34.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.34.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.34.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.254.keytab.
Entry for principal kudu/127.12.34.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.254.keytab.
I20260926 15:09:27.293736 12427 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8uoJGv/build/asan/bin/kudu
/tmp/dist-test-task8uoJGv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435318278681-12427-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435318278681-12427-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435318278681-12427-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.34.254:35279
--webserver_interface=127.12.34.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.254.keytab
--principal=kudu/127.12.34.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.34.212:44953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.34.254:35279
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435318278681-12427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435318278681-12427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435318278681-12427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:09:27.607462 18028 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:09:27.607710 18028 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:09:27.607864 18028 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:09:27.614842 18028 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:09:27.614933 18028 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:09:27.615041 18028 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:09:27.615125 18028 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:09:27.615188 18028 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:09:27.624509 18028 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.34.212:44953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435318278681-12427-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435318278681-12427-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.34.254:35279
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.254.keytab
--principal=kudu/127.12.34.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.34.254:35279
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.34.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435318278681-12427-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:09:27.625725 18028 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:09:27.626924 18028 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:09:27.638394 18033 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:09:27.638319 18034 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:09:27.638872 18036 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:09:27.638947 18028 server_base.cc:1034] running on GCE node
I20260926 15:09:27.639745 18028 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:09:27.641127 18028 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:09:27.642382 18028 hybrid_clock.cc:648] HybridClock initialized: now 1790435367642325 us; error 47 us; skew 500 ppm
I20260926 15:09:27.642736 18028 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:09:27 dist-test-slave-865v krb5kdc[18015](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435367, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:09:27.645797 18028 init.cc:377] Logged in from keytab as kudu/127.12.34.254@KRBTEST.COM (short username kudu)
I20260926 15:09:27.647222 18028 webserver.cc:460] Webserver started at http://127.12.34.254:46001/ using document root <none> and password file <none>
I20260926 15:09:27.647891 18028 fs_manager.cc:359] Metadata directory not provided
I20260926 15:09:27.648037 18028 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:09:27.648343 18028 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:09:27.651131 18028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435318278681-12427-0/minicluster-data/master-0/data/instance:
uuid: "862f7ba809ae46c0a22fdcf333f8f419"
format_stamp: "Formatted at 2026-09-26 15:09:27 on dist-test-slave-865v"
server_key: "de32b25dc7a04509ef90cb4a40a98c1f"
server_key_iv: "929bffb5f08b291a8bded0fa606757b2"
server_key_version: "encryptionkey@0"
I20260926 15:09:27.651911 18028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435318278681-12427-0/minicluster-data/master-0/wal/instance:
uuid: "862f7ba809ae46c0a22fdcf333f8f419"
format_stamp: "Formatted at 2026-09-26 15:09:27 on dist-test-slave-865v"
server_key: "de32b25dc7a04509ef90cb4a40a98c1f"
server_key_iv: "929bffb5f08b291a8bded0fa606757b2"
server_key_version: "encryptionkey@0"
I20260926 15:09:27.656685 18028 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260926 15:09:27.660025 18043 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:09:27.661283 18028 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 15:09:27.661458 18028 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435318278681-12427-0/minicluster-data/master-0/data,/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435318278681-12427-0/minicluster-data/master-0/wal
uuid: "862f7ba809ae46c0a22fdcf333f8f419"
format_stamp: "Formatted at 2026-09-26 15:09:27 on dist-test-slave-865v"
server_key: "de32b25dc7a04509ef90cb4a40a98c1f"
server_key_iv: "929bffb5f08b291a8bded0fa606757b2"
server_key_version: "encryptionkey@0"
I20260926 15:09:27.661643 18028 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435318278681-12427-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435318278681-12427-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435318278681-12427-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:09:27.694088 18028 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:09:27.697893 18028 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:09:27.698174 18028 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:09:27.716279 18028 rpc_server.cc:307] RPC server started. Bound to: 127.12.34.254:35279
I20260926 15:09:27.716315 18094 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.34.254:35279 every 8 connection(s)
I20260926 15:09:27.717643 18028 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
I20260926 15:09:27.721027 18095 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:09:27.722700 12427 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8uoJGv/build/asan/bin/kudu as pid 18028
I20260926 15:09:27.722915 12427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435318278681-12427-0/minicluster-data/master-0/wal/instance
I20260926 15:09:27.723508 12427 external_mini_cluster.cc:1428] Setting key f4189877ed8a6f23c5bae1606a83a635
I20260926 15:09:27.731228 18095 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 862f7ba809ae46c0a22fdcf333f8f419: Bootstrap starting.
I20260926 15:09:27.734149 18095 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 862f7ba809ae46c0a22fdcf333f8f419: Neither blocks nor log segments found. Creating new log.
I20260926 15:09:27.735059 18095 log.cc:824] T 00000000000000000000000000000000 P 862f7ba809ae46c0a22fdcf333f8f419: Log is configured to *not* fsync() on all Append() calls
Sep 26 15:09:27 dist-test-slave-865v krb5kdc[18015](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435367, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.34.254@KRBTEST.COM
I20260926 15:09:27.737926 18095 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 862f7ba809ae46c0a22fdcf333f8f419: No bootstrap required, opened a new log
I20260926 15:09:27.744421 18098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:27.725354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45912 (local address 127.12.34.254:35279)
0926 15:09:27.726040 (+ 686us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:27.726054 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:27.726091 (+ 37us) server_negotiation.cc:407] Connection header received
0926 15:09:27.727689 (+ 1598us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:27.727708 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:27.728045 (+ 337us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:27.728415 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:09:27.729846 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:27.731049 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:27.732306 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:27.732967 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:27.733186 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:27.737213 (+ 4027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:09:27.737223 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:09:27.737240 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:09:27.737286 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:09:27.740536 (+ 3250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:27.741282 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:27.741289 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:27.741295 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:27.741454 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:27.741872 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:27.741877 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:27.741891 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:27.742362 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:09:27.742675 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0926 15:09:27.743100 (+ 425us) server_negotiation.cc:299] Negotiation successful
0926 15:09:27.743489 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":507,"thread_start_us":304,"threads_started":1}
I20260926 15:09:27.745762 18095 raft_consensus.cc:348] T 00000000000000000000000000000000 P 862f7ba809ae46c0a22fdcf333f8f419 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "862f7ba809ae46c0a22fdcf333f8f419" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 35279 } }
I20260926 15:09:27.746141 18095 raft_consensus.cc:374] T 00000000000000000000000000000000 P 862f7ba809ae46c0a22fdcf333f8f419 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:09:27.746268 18095 raft_consensus.cc:729] T 00000000000000000000000000000000 P 862f7ba809ae46c0a22fdcf333f8f419 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 862f7ba809ae46c0a22fdcf333f8f419, State: Initialized, Role: FOLLOWER
I20260926 15:09:27.746774 18095 consensus_queue.cc:260] T 00000000000000000000000000000000 P 862f7ba809ae46c0a22fdcf333f8f419 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "862f7ba809ae46c0a22fdcf333f8f419" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 35279 } }
I20260926 15:09:27.747030 18095 raft_consensus.cc:388] T 00000000000000000000000000000000 P 862f7ba809ae46c0a22fdcf333f8f419 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 15:09:27.747171 18095 raft_consensus.cc:482] T 00000000000000000000000000000000 P 862f7ba809ae46c0a22fdcf333f8f419 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 15:09:27.747372 18095 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 862f7ba809ae46c0a22fdcf333f8f419 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:09:27.749584 18095 raft_consensus.cc:504] T 00000000000000000000000000000000 P 862f7ba809ae46c0a22fdcf333f8f419 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "862f7ba809ae46c0a22fdcf333f8f419" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 35279 } }
I20260926 15:09:27.750126 18095 leader_election.cc:302] T 00000000000000000000000000000000 P 862f7ba809ae46c0a22fdcf333f8f419 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 862f7ba809ae46c0a22fdcf333f8f419; no voters:
I20260926 15:09:27.750773 18095 leader_election.cc:288] T 00000000000000000000000000000000 P 862f7ba809ae46c0a22fdcf333f8f419 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 15:09:27.750893 18100 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 862f7ba809ae46c0a22fdcf333f8f419 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:09:27.751971 18100 raft_consensus.cc:686] T 00000000000000000000000000000000 P 862f7ba809ae46c0a22fdcf333f8f419 [term 1 LEADER]: Becoming Leader. State: Replica: 862f7ba809ae46c0a22fdcf333f8f419, State: Running, Role: LEADER
I20260926 15:09:27.752461 18100 consensus_queue.cc:237] T 00000000000000000000000000000000 P 862f7ba809ae46c0a22fdcf333f8f419 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "862f7ba809ae46c0a22fdcf333f8f419" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 35279 } }
I20260926 15:09:27.752581 18095 sys_catalog.cc:564] T 00000000000000000000000000000000 P 862f7ba809ae46c0a22fdcf333f8f419 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:09:27.755637 18102 sys_catalog.cc:455] T 00000000000000000000000000000000 P 862f7ba809ae46c0a22fdcf333f8f419 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 862f7ba809ae46c0a22fdcf333f8f419. Latest consensus state: current_term: 1 leader_uuid: "862f7ba809ae46c0a22fdcf333f8f419" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "862f7ba809ae46c0a22fdcf333f8f419" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 35279 } } }
I20260926 15:09:27.756114 18102 sys_catalog.cc:458] T 00000000000000000000000000000000 P 862f7ba809ae46c0a22fdcf333f8f419 [sys.catalog]: This master's current role is: LEADER
I20260926 15:09:27.756022 18101 sys_catalog.cc:455] T 00000000000000000000000000000000 P 862f7ba809ae46c0a22fdcf333f8f419 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "862f7ba809ae46c0a22fdcf333f8f419" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "862f7ba809ae46c0a22fdcf333f8f419" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 35279 } } }
I20260926 15:09:27.756316 18101 sys_catalog.cc:458] T 00000000000000000000000000000000 P 862f7ba809ae46c0a22fdcf333f8f419 [sys.catalog]: This master's current role is: LEADER
I20260926 15:09:27.757910 18108 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:09:27.762754 18108 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 15:09:27.771462 18108 catalog_manager.cc:1347] Generated new cluster ID: a7bbb1771a834cf8a53e7f2a43933e59
I20260926 15:09:27.771574 18108 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 15:09:27.798448 18108 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 15:09:27.799796 18108 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:09:27.808885 18108 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 862f7ba809ae46c0a22fdcf333f8f419: Generated new TSK 0
I20260926 15:09:27.809734 18108 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.34.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.34.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.34.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.193.keytab.
Entry for principal kudu/127.12.34.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.193.keytab.
I20260926 15:09:27.872427 12427 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8uoJGv/build/asan/bin/kudu
/tmp/dist-test-task8uoJGv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435318278681-12427-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435318278681-12427-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435318278681-12427-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435318278681-12427-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.34.193:0
--local_ip_for_outbound_sockets=127.12.34.193
--webserver_interface=127.12.34.193
--webserver_port=0
--tserver_master_addrs=127.12.34.254:35279
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.193.keytab
--principal=kudu/127.12.34.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.34.212:44953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435318278681-12427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435318278681-12427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435318278681-12427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:09:28.203579 18121 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:09:28.203882 18121 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:09:28.204061 18121 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:09:28.211967 18121 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:09:28.212134 18121 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:09:28.212334 18121 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.34.193
I20260926 15:09:28.221724 18121 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.34.212:44953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435318278681-12427-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435318278681-12427-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.193.keytab
--principal=kudu/127.12.34.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.34.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435318278681-12427-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.34.193
--webserver_port=0
--tserver_master_addrs=127.12.34.254:35279
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.34.193
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435318278681-12427-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:09:28.222985 18121 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:09:28.224246 18121 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:09:28.236883 18126 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:09:28.236992 18129 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:09:28.236955 18127 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:09:28.237936 18121 server_base.cc:1034] running on GCE node
I20260926 15:09:28.238561 18121 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:09:28.239519 18121 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:09:28.240777 18121 hybrid_clock.cc:648] HybridClock initialized: now 1790435368240748 us; error 51 us; skew 500 ppm
I20260926 15:09:28.241149 18121 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:09:28 dist-test-slave-865v krb5kdc[18015](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435368, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:09:28.245265 18121 init.cc:377] Logged in from keytab as kudu/127.12.34.193@KRBTEST.COM (short username kudu)
I20260926 15:09:28.246989 18121 webserver.cc:460] Webserver started at http://127.12.34.193:43799/ using document root <none> and password file <none>
I20260926 15:09:28.247704 18121 fs_manager.cc:359] Metadata directory not provided
I20260926 15:09:28.247860 18121 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:09:28.248181 18121 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:09:28.251057 18121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435318278681-12427-0/minicluster-data/ts-0/data/instance:
uuid: "969e6501f7ad4967b3a11f434cb8a217"
format_stamp: "Formatted at 2026-09-26 15:09:28 on dist-test-slave-865v"
server_key: "2f2cccef262267e2c1bbb597b4a3af8d"
server_key_iv: "b2e7d85b4dfdc048e715df7a3e28ede2"
server_key_version: "encryptionkey@0"
I20260926 15:09:28.251848 18121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435318278681-12427-0/minicluster-data/ts-0/wal/instance:
uuid: "969e6501f7ad4967b3a11f434cb8a217"
format_stamp: "Formatted at 2026-09-26 15:09:28 on dist-test-slave-865v"
server_key: "2f2cccef262267e2c1bbb597b4a3af8d"
server_key_iv: "b2e7d85b4dfdc048e715df7a3e28ede2"
server_key_version: "encryptionkey@0"
I20260926 15:09:28.256815 18121 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260926 15:09:28.260488 18136 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:09:28.261904 18121 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 15:09:28.262089 18121 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435318278681-12427-0/minicluster-data/ts-0/data,/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435318278681-12427-0/minicluster-data/ts-0/wal
uuid: "969e6501f7ad4967b3a11f434cb8a217"
format_stamp: "Formatted at 2026-09-26 15:09:28 on dist-test-slave-865v"
server_key: "2f2cccef262267e2c1bbb597b4a3af8d"
server_key_iv: "b2e7d85b4dfdc048e715df7a3e28ede2"
server_key_version: "encryptionkey@0"
I20260926 15:09:28.262332 18121 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435318278681-12427-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435318278681-12427-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435318278681-12427-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:09:28.277205 18121 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:09:28.280936 18121 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:09:28.281263 18121 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:09:28.282161 18121 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:09:28.283427 18121 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:09:28.283533 18121 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:09:28.283690 18121 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:09:28.283768 18121 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:09:28.316999 18121 rpc_server.cc:307] RPC server started. Bound to: 127.12.34.193:46687
I20260926 15:09:28.317142 18248 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.34.193:46687 every 8 connection(s)
I20260926 15:09:28.318327 18121 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435318278681-12427-0/minicluster-data/ts-0/data/info.pb
I20260926 15:09:28.324400 12427 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8uoJGv/build/asan/bin/kudu as pid 18121
I20260926 15:09:28.324594 12427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435318278681-12427-0/minicluster-data/ts-0/wal/instance
I20260926 15:09:28.325255 12427 external_mini_cluster.cc:1428] Setting key 0506e6c50c084dc8eb919fbd9e8985a7
Sep 26 15:09:28 dist-test-slave-865v krb5kdc[18015](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435368, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.193@KRBTEST.COM for kudu/127.12.34.254@KRBTEST.COM
I20260926 15:09:28.339675 18251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:28.320862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.193:51641 (local address 127.12.34.254:35279)
0926 15:09:28.321324 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:28.321328 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:28.322244 (+ 916us) server_negotiation.cc:407] Connection header received
0926 15:09:28.324013 (+ 1769us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:28.324019 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:28.324104 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:28.324223 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:09:28.325844 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:28.326671 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:28.329418 (+ 2747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:28.329821 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:28.329914 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:28.333896 (+ 3982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:09:28.333901 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:09:28.333904 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:09:28.333932 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:09:28.336408 (+ 2476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:28.337130 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:28.337139 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:28.337142 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:28.337250 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:28.337676 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:28.337681 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:28.337684 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:28.337922 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:09:28.338086 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0926 15:09:28.339275 (+ 1189us) server_negotiation.cc:299] Negotiation successful
0926 15:09:28.339423 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":276,"threads_started":1}
I20260926 15:09:28.340768 18252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:28.321396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:35279 (local address 127.12.34.193:51641)
0926 15:09:28.322091 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0926 15:09:28.322131 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 15:09:28.323720 (+ 1589us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:09:28.324441 (+ 721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:09:28.324449 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:09:28.325038 (+ 589us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:09:28.325678 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:28.325695 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:28.327110 (+ 1415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:28.327117 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:28.329276 (+ 2159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:28.329294 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:28.330051 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:28.330060 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:28.330376 (+ 316us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:28.331099 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:09:28.331133 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:09:28.333661 (+ 2528us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:09:28.336564 (+ 2903us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:09:28.336573 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:09:28.336586 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:09:28.336983 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:09:28.337362 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:09:28.337368 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:09:28.337372 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:09:28.337554 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:09:28.338093 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:09:28.338104 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:09:28.338538 (+ 434us) client_negotiation.cc:769] Sending connection context
0926 15:09:28.338814 (+ 276us) client_negotiation.cc:240] Negotiation successful
0926 15:09:28.339342 (+ 528us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":284,"threads_started":1}
I20260926 15:09:28.342630 18249 heartbeater.cc:347] Connected to a master server at 127.12.34.254:35279
I20260926 15:09:28.342928 18249 heartbeater.cc:464] Registering TS with master...
I20260926 15:09:28.343631 18249 heartbeater.cc:511] Master 127.12.34.254:35279 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.12.34.194@KRBTEST.COM; defaulting to no policy
I20260926 15:09:28.345754 18060 ts_manager.cc:194] Registered new tserver with Master: 969e6501f7ad4967b3a11f434cb8a217 (127.12.34.193:46687)
I20260926 15:09:28.347184 18060 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.34.193@KRBTEST.COM'} at 127.12.34.193:51641
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.34.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.34.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.194.keytab.
Entry for principal kudu/127.12.34.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.194.keytab.
I20260926 15:09:28.372224 12427 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8uoJGv/build/asan/bin/kudu
/tmp/dist-test-task8uoJGv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435318278681-12427-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435318278681-12427-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435318278681-12427-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435318278681-12427-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.34.194:0
--local_ip_for_outbound_sockets=127.12.34.194
--webserver_interface=127.12.34.194
--webserver_port=0
--tserver_master_addrs=127.12.34.254:35279
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.194.keytab
--principal=kudu/127.12.34.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.34.212:44953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435318278681-12427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435318278681-12427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435318278681-12427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:09:28.687273 18255 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:09:28.687584 18255 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:09:28.687753 18255 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:09:28.694769 18255 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:09:28.694918 18255 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:09:28.695102 18255 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.34.194
I20260926 15:09:28.704465 18255 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.34.212:44953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435318278681-12427-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435318278681-12427-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.194.keytab
--principal=kudu/127.12.34.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.34.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435318278681-12427-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.34.194
--webserver_port=0
--tserver_master_addrs=127.12.34.254:35279
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.34.194
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435318278681-12427-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:09:28.705741 18255 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:09:28.706945 18255 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:09:28.719619 18261 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:09:28.720027 18260 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:09:28.720064 18263 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:09:28.720541 18255 server_base.cc:1034] running on GCE node
I20260926 15:09:28.721186 18255 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:09:28.722219 18255 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:09:28.723456 18255 hybrid_clock.cc:648] HybridClock initialized: now 1790435368723405 us; error 42 us; skew 500 ppm
I20260926 15:09:28.723784 18255 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:09:28 dist-test-slave-865v krb5kdc[18015](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435368, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:09:28.732221 18255 init.cc:377] Logged in from keytab as kudu/127.12.34.194@KRBTEST.COM (short username kudu)
I20260926 15:09:28.733800 18255 webserver.cc:460] Webserver started at http://127.12.34.194:35413/ using document root <none> and password file <none>
I20260926 15:09:28.734462 18255 fs_manager.cc:359] Metadata directory not provided
I20260926 15:09:28.734607 18255 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:09:28.734906 18255 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:09:28.737744 18255 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435318278681-12427-0/minicluster-data/ts-1/data/instance:
uuid: "581413c3ffb9494798349b251b25cc69"
format_stamp: "Formatted at 2026-09-26 15:09:28 on dist-test-slave-865v"
server_key: "cebc7a834e011bd0251e75ee67968456"
server_key_iv: "52196f40cc1ea74d5d65e82054bfdd6f"
server_key_version: "encryptionkey@0"
I20260926 15:09:28.738488 18255 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435318278681-12427-0/minicluster-data/ts-1/wal/instance:
uuid: "581413c3ffb9494798349b251b25cc69"
format_stamp: "Formatted at 2026-09-26 15:09:28 on dist-test-slave-865v"
server_key: "cebc7a834e011bd0251e75ee67968456"
server_key_iv: "52196f40cc1ea74d5d65e82054bfdd6f"
server_key_version: "encryptionkey@0"
I20260926 15:09:28.743721 18255 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 15:09:28.747202 18270 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:09:28.748564 18255 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 15:09:28.748741 18255 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435318278681-12427-0/minicluster-data/ts-1/data,/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435318278681-12427-0/minicluster-data/ts-1/wal
uuid: "581413c3ffb9494798349b251b25cc69"
format_stamp: "Formatted at 2026-09-26 15:09:28 on dist-test-slave-865v"
server_key: "cebc7a834e011bd0251e75ee67968456"
server_key_iv: "52196f40cc1ea74d5d65e82054bfdd6f"
server_key_version: "encryptionkey@0"
I20260926 15:09:28.748952 18255 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435318278681-12427-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435318278681-12427-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435318278681-12427-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:09:28.765379 18255 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:09:28.769459 18255 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:09:28.769745 18255 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:09:28.770674 18255 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:09:28.771967 18255 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:09:28.772074 18255 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:09:28.772233 18255 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:09:28.772310 18255 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:09:28.807331 18255 rpc_server.cc:307] RPC server started. Bound to: 127.12.34.194:44301
I20260926 15:09:28.807394 18382 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.34.194:44301 every 8 connection(s)
I20260926 15:09:28.808683 18255 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435318278681-12427-0/minicluster-data/ts-1/data/info.pb
I20260926 15:09:28.810366 12427 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8uoJGv/build/asan/bin/kudu as pid 18255
I20260926 15:09:28.810546 12427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435318278681-12427-0/minicluster-data/ts-1/wal/instance
I20260926 15:09:28.811125 12427 external_mini_cluster.cc:1428] Setting key e49650a9642b31fa0f345fc44dbcae7c
Sep 26 15:09:28 dist-test-slave-865v krb5kdc[18015](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435368, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.194@KRBTEST.COM for kudu/127.12.34.254@KRBTEST.COM
I20260926 15:09:28.830238 18251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:28.811484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.194:49191 (local address 127.12.34.254:35279)
0926 15:09:28.811659 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:28.811687 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:28.813039 (+ 1352us) server_negotiation.cc:407] Connection header received
0926 15:09:28.814886 (+ 1847us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:28.814891 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:28.814976 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:28.815083 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:09:28.816492 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:28.817257 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:28.818736 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:28.819154 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:28.819288 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:28.823131 (+ 3843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:09:28.823141 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:09:28.823149 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:09:28.823193 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:09:28.826966 (+ 3773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:28.827741 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:28.827748 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:28.827753 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:28.827901 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:28.828323 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:28.828330 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:28.828335 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:28.828736 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:09:28.828901 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0926 15:09:28.829587 (+ 686us) server_negotiation.cc:299] Negotiation successful
0926 15:09:28.829824 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 15:09:28.830924 18385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:28.812046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:35279 (local address 127.12.34.194:49191)
0926 15:09:28.812884 (+ 838us) negotiation.cc:107] Waiting for socket to connect
0926 15:09:28.812939 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0926 15:09:28.814611 (+ 1672us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:09:28.815305 (+ 694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:09:28.815313 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:09:28.815796 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:09:28.816317 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:28.816343 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:28.817411 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:28.817422 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:28.818585 (+ 1163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:28.818595 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:28.819376 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:28.819383 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:28.819596 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:28.820317 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:09:28.820339 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:09:28.822879 (+ 2540us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:09:28.827114 (+ 4235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:09:28.827121 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:09:28.827133 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:09:28.827600 (+ 467us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:09:28.828028 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:09:28.828032 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:09:28.828034 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:09:28.828203 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:09:28.828862 (+ 659us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:09:28.828868 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:09:28.829247 (+ 379us) client_negotiation.cc:769] Sending connection context
0926 15:09:28.829477 (+ 230us) client_negotiation.cc:240] Negotiation successful
0926 15:09:28.829831 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":559,"thread_start_us":351,"threads_started":1}
WARNING: no policy specified for kudu/127.12.34.195@KRBTEST.COM; defaulting to no policy
I20260926 15:09:28.832582 18383 heartbeater.cc:347] Connected to a master server at 127.12.34.254:35279
I20260926 15:09:28.832983 18383 heartbeater.cc:464] Registering TS with master...
I20260926 15:09:28.833747 18383 heartbeater.cc:511] Master 127.12.34.254:35279 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.34.195@KRBTEST.COM" created.
I20260926 15:09:28.835419 18060 ts_manager.cc:194] Registered new tserver with Master: 581413c3ffb9494798349b251b25cc69 (127.12.34.194:44301)
I20260926 15:09:28.836228 18060 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.34.194@KRBTEST.COM'} at 127.12.34.194:49191
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.34.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.195.keytab.
Entry for principal kudu/127.12.34.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.195.keytab.
I20260926 15:09:28.858877 12427 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8uoJGv/build/asan/bin/kudu
/tmp/dist-test-task8uoJGv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435318278681-12427-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435318278681-12427-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435318278681-12427-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435318278681-12427-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.34.195:0
--local_ip_for_outbound_sockets=127.12.34.195
--webserver_interface=127.12.34.195
--webserver_port=0
--tserver_master_addrs=127.12.34.254:35279
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.195.keytab
--principal=kudu/127.12.34.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.34.212:44953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435318278681-12427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435318278681-12427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435318278681-12427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:09:29.171525 18388 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:09:29.171819 18388 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:09:29.172118 18388 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:09:29.179221 18388 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:09:29.179390 18388 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:09:29.179561 18388 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.34.195
I20260926 15:09:29.188937 18388 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.34.212:44953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435318278681-12427-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435318278681-12427-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.195.keytab
--principal=kudu/127.12.34.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.34.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435318278681-12427-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.34.195
--webserver_port=0
--tserver_master_addrs=127.12.34.254:35279
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.34.195
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435318278681-12427-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:09:29.190152 18388 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:09:29.191382 18388 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:09:29.203765 18393 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:09:29.204088 18396 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:09:29.204154 18394 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:09:29.205183 18388 server_base.cc:1034] running on GCE node
I20260926 15:09:29.205747 18388 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:09:29.206650 18388 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:09:29.207880 18388 hybrid_clock.cc:648] HybridClock initialized: now 1790435369207846 us; error 60 us; skew 500 ppm
I20260926 15:09:29.208235 18388 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:09:29 dist-test-slave-865v krb5kdc[18015](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435369, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:09:29.211320 18388 init.cc:377] Logged in from keytab as kudu/127.12.34.195@KRBTEST.COM (short username kudu)
I20260926 15:09:29.212672 18388 webserver.cc:460] Webserver started at http://127.12.34.195:38983/ using document root <none> and password file <none>
I20260926 15:09:29.213398 18388 fs_manager.cc:359] Metadata directory not provided
I20260926 15:09:29.213537 18388 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:09:29.213830 18388 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:09:29.216500 18388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435318278681-12427-0/minicluster-data/ts-2/data/instance:
uuid: "2ac5c42262984f488ce1f33fca268874"
format_stamp: "Formatted at 2026-09-26 15:09:29 on dist-test-slave-865v"
server_key: "ee50f87898c0bc7334f72de018a4745d"
server_key_iv: "9770dec120d75307c6824b6db25b7a73"
server_key_version: "encryptionkey@0"
I20260926 15:09:29.217329 18388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435318278681-12427-0/minicluster-data/ts-2/wal/instance:
uuid: "2ac5c42262984f488ce1f33fca268874"
format_stamp: "Formatted at 2026-09-26 15:09:29 on dist-test-slave-865v"
server_key: "ee50f87898c0bc7334f72de018a4745d"
server_key_iv: "9770dec120d75307c6824b6db25b7a73"
server_key_version: "encryptionkey@0"
I20260926 15:09:29.222170 18388 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 15:09:29.225703 18403 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:09:29.227085 18388 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 15:09:29.227257 18388 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435318278681-12427-0/minicluster-data/ts-2/data,/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435318278681-12427-0/minicluster-data/ts-2/wal
uuid: "2ac5c42262984f488ce1f33fca268874"
format_stamp: "Formatted at 2026-09-26 15:09:29 on dist-test-slave-865v"
server_key: "ee50f87898c0bc7334f72de018a4745d"
server_key_iv: "9770dec120d75307c6824b6db25b7a73"
server_key_version: "encryptionkey@0"
I20260926 15:09:29.227443 18388 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435318278681-12427-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435318278681-12427-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435318278681-12427-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:09:29.247743 18388 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:09:29.252043 18388 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:09:29.252403 18388 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:09:29.253400 18388 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:09:29.254782 18388 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:09:29.254855 18388 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:09:29.255003 18388 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:09:29.255064 18388 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:09:29.286741 18388 rpc_server.cc:307] RPC server started. Bound to: 127.12.34.195:38041
I20260926 15:09:29.286793 18515 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.34.195:38041 every 8 connection(s)
I20260926 15:09:29.287971 18388 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435318278681-12427-0/minicluster-data/ts-2/data/info.pb
I20260926 15:09:29.297595 12427 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8uoJGv/build/asan/bin/kudu as pid 18388
I20260926 15:09:29.297781 12427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435318278681-12427-0/minicluster-data/ts-2/wal/instance
I20260926 15:09:29.298202 12427 external_mini_cluster.cc:1428] Setting key c47ad252b2ea96591edd07ca328e5e77
Sep 26 15:09:29 dist-test-slave-865v krb5kdc[18015](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435369, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.195@KRBTEST.COM for kudu/127.12.34.254@KRBTEST.COM
I20260926 15:09:29.307399 18251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:29.290361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.195:57565 (local address 127.12.34.254:35279)
0926 15:09:29.290542 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:29.290546 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:29.291656 (+ 1110us) server_negotiation.cc:407] Connection header received
0926 15:09:29.293496 (+ 1840us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:29.293501 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:29.293588 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:29.293700 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:09:29.295068 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:29.295804 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:29.297348 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:29.297843 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:29.298003 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:29.301980 (+ 3977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:09:29.301985 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:09:29.301989 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:09:29.302018 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:09:29.304615 (+ 2597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:29.305383 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:29.305387 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:29.305390 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:29.305516 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:29.305890 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:29.305893 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:29.305895 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:29.306150 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:09:29.306263 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0926 15:09:29.306985 (+ 722us) server_negotiation.cc:299] Negotiation successful
0926 15:09:29.307142 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 15:09:29.308324 18518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:29.290801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:35279 (local address 127.12.34.195:57565)
0926 15:09:29.291499 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0926 15:09:29.291533 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 15:09:29.293231 (+ 1698us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:09:29.293914 (+ 683us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:09:29.293922 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:09:29.294386 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:09:29.294903 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:29.294919 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:29.295953 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:29.295960 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:29.297200 (+ 1240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:29.297212 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:29.298071 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:29.298080 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:29.298361 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:29.299047 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:09:29.299068 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:09:29.301752 (+ 2684us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:09:29.304750 (+ 2998us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:09:29.304757 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:09:29.304768 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:09:29.305260 (+ 492us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:09:29.305614 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:09:29.305617 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:09:29.305620 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:09:29.305791 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:09:29.306296 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:09:29.306302 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:09:29.306638 (+ 336us) client_negotiation.cc:769] Sending connection context
0926 15:09:29.306862 (+ 224us) client_negotiation.cc:240] Negotiation successful
0926 15:09:29.307244 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":288,"threads_started":1}
I20260926 15:09:29.309741 18516 heartbeater.cc:347] Connected to a master server at 127.12.34.254:35279
I20260926 15:09:29.310019 18516 heartbeater.cc:464] Registering TS with master...
I20260926 15:09:29.310660 18516 heartbeater.cc:511] Master 127.12.34.254:35279 requested a full tablet report, sending...
I20260926 15:09:29.312215 18060 ts_manager.cc:194] Registered new tserver with Master: 2ac5c42262984f488ce1f33fca268874 (127.12.34.195:38041)
I20260926 15:09:29.313241 18060 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.34.195@KRBTEST.COM'} at 127.12.34.195:57565
I20260926 15:09:29.314051 12427 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435318278681-12427-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435318278681-12427-0/krb5kdc/test-admin.keytab.
Sep 26 15:09:29 dist-test-slave-865v krb5kdc[18015](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435369, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:09:29.341977 12427 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260926 15:09:29.346804 18523 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:09:29.344507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50976 (local address 127.12.34.193:46687)
0926 15:09:29.345156 (+ 649us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:29.345164 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:29.345189 (+ 25us) server_negotiation.cc:407] Connection header received
0926 15:09:29.345246 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:29.345253 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:29.345468 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:29.345595 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:09:29.346441 (+ 846us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50976: BlockingRecv error: recv got EOF from 127.0.0.1:50976 (error 108)
Metrics: {"server-negotiator.queue_time_us":457,"thread_start_us":310,"threads_started":1}
W20260926 15:09:29.347154 12427 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435318278681-12427-0/krb5kdc/krb5cc' of type FILE
Sep 26 15:09:29 dist-test-slave-865v krb5kdc[18015](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435369, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:09:29.348897 12427 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 15:09:29 dist-test-slave-865v krb5kdc[18015](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435369, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:09:29.350342 12427 init.cc:295] Successfully reacquired a new kerberos TGT
I20260926 15:09:29.350690 18249 heartbeater.cc:503] Master 127.12.34.254:35279 was elected leader, sending a full tablet report...
Sep 26 15:09:29 dist-test-slave-865v krb5kdc[18015](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435369, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.34.193@KRBTEST.COM
I20260926 15:09:29.365872 18523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:29.352087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50988 (local address 127.12.34.193:46687)
0926 15:09:29.352295 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:29.352301 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:29.352813 (+ 512us) server_negotiation.cc:407] Connection header received
0926 15:09:29.352857 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:29.352860 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:29.352912 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:29.352979 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:09:29.354051 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:29.354968 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:29.356188 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:29.356739 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:29.356847 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:29.360398 (+ 3551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:09:29.360409 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:09:29.360422 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:09:29.360450 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:09:29.363230 (+ 2780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:29.363887 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:29.363899 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:29.363902 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:29.364010 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:29.364388 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:29.364393 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:29.364396 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:29.364721 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:09:29.365068 (+ 347us) server_negotiation.cc:1035] Waiting for connection context
0926 15:09:29.365377 (+ 309us) server_negotiation.cc:299] Negotiation successful
0926 15:09:29.365613 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
W20260926 15:09:29.371389 18523 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:09:29.369857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50996 (local address 127.12.34.193:46687)
0926 15:09:29.370045 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:29.370051 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:29.370221 (+ 170us) server_negotiation.cc:407] Connection header received
0926 15:09:29.370330 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:29.370333 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:29.370406 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:29.370478 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:09:29.371148 (+ 670us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50996: BlockingRecv error: recv got EOF from 127.0.0.1:50996 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260926 15:09:29.371763 12427 init.cc:244] Error while opening credential cache '/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435318278681-12427-0/krb5kdc/krb5cc' of type FILE: Runtime error: Bad format in credentials cache (filename: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435318278681-12427-0/krb5kdc/krb5cc)
Sep 26 15:09:29 dist-test-slave-865v krb5kdc[18015](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435369, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:09:29.373274 12427 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 15:09:29 dist-test-slave-865v krb5kdc[18015](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435369, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:09:29.374657 12427 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 15:09:29 dist-test-slave-865v krb5kdc[18015](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435369, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.34.193@KRBTEST.COM
I20260926 15:09:29.388160 18523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:29.375855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51008 (local address 127.12.34.193:46687)
0926 15:09:29.376071 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:29.376077 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:29.376312 (+ 235us) server_negotiation.cc:407] Connection header received
0926 15:09:29.376482 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:29.376490 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:29.376561 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:29.376660 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:09:29.377570 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:29.378316 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:29.379542 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:29.379970 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:29.380104 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:29.383448 (+ 3344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:09:29.383459 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:09:29.383464 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:09:29.383504 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:09:29.385841 (+ 2337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:29.386445 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:29.386451 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:29.386460 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:29.386580 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:29.386939 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:29.386946 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:29.386949 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:29.387184 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:09:29.387443 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0926 15:09:29.387729 (+ 286us) server_negotiation.cc:299] Negotiation successful
0926 15:09:29.387920 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
W20260926 15:09:29.391785 18523 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:09:29.390486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51010 (local address 127.12.34.193:46687)
0926 15:09:29.390655 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:29.390658 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:29.390736 (+ 78us) server_negotiation.cc:407] Connection header received
0926 15:09:29.390838 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:29.390841 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:29.390906 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:29.390977 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:09:29.391589 (+ 612us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51010: BlockingRecv error: recv got EOF from 127.0.0.1:51010 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260926 15:09:29.391914 12427 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790435318278681-12427-0/krb5kdc/krb5cc' of type FILE
Sep 26 15:09:29 dist-test-slave-865v krb5kdc[18015](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435369, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:09:29.393450 12427 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 15:09:29 dist-test-slave-865v krb5kdc[18015](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435369, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:09:29.394701 12427 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 15:09:29 dist-test-slave-865v krb5kdc[18015](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435369, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.34.193@KRBTEST.COM
I20260926 15:09:29.408187 18523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:29.395854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51012 (local address 127.12.34.193:46687)
0926 15:09:29.396062 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:29.396068 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:29.396340 (+ 272us) server_negotiation.cc:407] Connection header received
0926 15:09:29.396513 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:29.396518 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:29.396579 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:29.396674 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:09:29.397621 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:29.398293 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:29.399441 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:29.399871 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:29.400005 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:29.403396 (+ 3391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:09:29.403401 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:09:29.403406 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:09:29.403449 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:09:29.405830 (+ 2381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:29.406425 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:29.406432 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:29.406435 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:29.406578 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:29.406935 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:29.406942 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:29.406945 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:29.407168 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:09:29.407491 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0926 15:09:29.407720 (+ 229us) server_negotiation.cc:299] Negotiation successful
0926 15:09:29.407920 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 15:09:29.409749 12427 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 18121
I20260926 15:09:29.421420 18523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:29.410445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51022 (local address 127.12.34.193:46687)
0926 15:09:29.410674 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:29.410681 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:29.410946 (+ 265us) server_negotiation.cc:407] Connection header received
0926 15:09:29.411143 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:29.411148 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:29.411212 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:29.411311 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:09:29.412163 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:29.412855 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:29.414085 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:29.414514 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:29.414643 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:29.416516 (+ 1873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:09:29.416522 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:09:29.416527 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:09:29.416565 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:09:29.418915 (+ 2350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:29.419581 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:29.419588 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:29.419592 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:29.419721 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:29.420113 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:29.420120 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:29.420123 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:29.420353 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:09:29.420619 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0926 15:09:29.420911 (+ 292us) server_negotiation.cc:299] Negotiation successful
0926 15:09:29.421120 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 15:09:29.422037 18244 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:09:29.504228 12427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8uoJGv/build/asan/bin/kudu with pid 18121
I20260926 15:09:29.520336 12427 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 18255
Sep 26 15:09:29 dist-test-slave-865v krb5kdc[18015](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435369, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.34.194@KRBTEST.COM
I20260926 15:09:29.535535 18536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:29.521260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58134 (local address 127.12.34.194:44301)
0926 15:09:29.521811 (+ 551us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:29.521818 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:29.521843 (+ 25us) server_negotiation.cc:407] Connection header received
0926 15:09:29.521912 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:29.521918 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:29.522134 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:29.522283 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:09:29.523183 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:29.524219 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:29.525537 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:29.526061 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:29.526165 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:29.529838 (+ 3673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:09:29.529848 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:09:29.529863 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:09:29.529894 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:09:29.532737 (+ 2843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:29.533637 (+ 900us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:29.533643 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:29.533645 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:29.533761 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:29.534092 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:29.534095 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:29.534097 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:29.534425 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:09:29.534644 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0926 15:09:29.534969 (+ 325us) server_negotiation.cc:299] Negotiation successful
0926 15:09:29.535239 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":420,"thread_start_us":304,"threads_started":1}
I20260926 15:09:29.536813 18378 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:09:29.618271 12427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8uoJGv/build/asan/bin/kudu with pid 18255
I20260926 15:09:29.634125 12427 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 18388
Sep 26 15:09:29 dist-test-slave-865v krb5kdc[18015](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435369, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.34.195@KRBTEST.COM
I20260926 15:09:29.649319 18538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:29.635264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51380 (local address 127.12.34.195:38041)
0926 15:09:29.635910 (+ 646us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:29.635916 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:29.635945 (+ 29us) server_negotiation.cc:407] Connection header received
0926 15:09:29.636002 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:29.636008 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:29.636228 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:29.636355 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:09:29.637398 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:29.638352 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:29.639649 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:29.640157 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:29.640270 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:29.644055 (+ 3785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:09:29.644065 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:09:29.644078 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:09:29.644110 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:09:29.646889 (+ 2779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:29.647491 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:29.647498 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:29.647500 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:29.647609 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:29.647903 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:29.647905 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:29.647907 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:29.648200 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:09:29.648420 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0926 15:09:29.648595 (+ 175us) server_negotiation.cc:299] Negotiation successful
0926 15:09:29.648841 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":505,"thread_start_us":353,"threads_started":1}
I20260926 15:09:29.650628 18511 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:09:29.737730 12427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8uoJGv/build/asan/bin/kudu with pid 18388
I20260926 15:09:29.753824 12427 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 18028
I20260926 15:09:29.754985 18090 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:09:29.857014 12427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8uoJGv/build/asan/bin/kudu with pid 18028
2026-09-26T15:09:29Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3876 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 15:09:29 dist-test-slave-865v krb5kdc[18543](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 15:09:29 dist-test-slave-865v krb5kdc[18543](info): set up 2 sockets
Sep 26 15:09:29 dist-test-slave-865v krb5kdc[18543](info): commencing operation
krb5kdc: starting...
W20260926 15:09:30.981989 12427 mini_kdc.cc:121] Time spent starting KDC: real 1.084s user 0.001s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/krb5kdc/test-user.keytab.
Sep 26 15:09:31 dist-test-slave-865v krb5kdc[18543](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435371, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T15:09:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T15:09:31Z Disabled control of system clock
WARNING: no policy specified for oryx/127.12.34.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.34.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.34.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/krb5kdc/oryx_127.12.34.254.keytab.
Entry for principal oryx/127.12.34.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/krb5kdc/oryx_127.12.34.254.keytab.
I20260926 15:09:31.178766 12427 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8uoJGv/build/asan/bin/kudu
/tmp/dist-test-task8uoJGv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.34.254:42537
--webserver_interface=127.12.34.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/krb5kdc/oryx_127.12.34.254.keytab
--principal=oryx/127.12.34.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.34.212:35299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.34.254:42537
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:09:31.486644 18556 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:09:31.486896 18556 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:09:31.486989 18556 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:09:31.493852 18556 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:09:31.493950 18556 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:09:31.494004 18556 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:09:31.494112 18556 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:09:31.494176 18556 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:09:31.503508 18556 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.34.212:35299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.34.254:42537
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/krb5kdc/oryx_127.12.34.254.keytab
--principal=oryx/127.12.34.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.34.254:42537
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.34.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:09:31.504737 18556 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:09:31.505935 18556 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:09:31.516914 18561 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:09:31.516973 18562 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:09:31.516973 18564 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:09:31.518095 18556 server_base.cc:1034] running on GCE node
I20260926 15:09:31.518776 18556 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:09:31.520025 18556 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:09:31.521238 18556 hybrid_clock.cc:648] HybridClock initialized: now 1790435371521205 us; error 48 us; skew 500 ppm
I20260926 15:09:31.521601 18556 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:09:31 dist-test-slave-865v krb5kdc[18543](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435371, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.34.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:09:31.524570 18556 init.cc:377] Logged in from keytab as oryx/127.12.34.254@KRBTEST.COM (short username oryx)
I20260926 15:09:31.526048 18556 webserver.cc:460] Webserver started at http://127.12.34.254:33335/ using document root <none> and password file <none>
I20260926 15:09:31.526716 18556 fs_manager.cc:359] Metadata directory not provided
I20260926 15:09:31.526859 18556 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:09:31.527151 18556 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:09:31.530297 18556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/minicluster-data/master-0/data/instance:
uuid: "f8c077ce1f004166989faaf5668ef1e8"
format_stamp: "Formatted at 2026-09-26 15:09:31 on dist-test-slave-865v"
server_key: "449fb37a37492bc0d57118fc48681ed2"
server_key_iv: "7e08b4583b7bf4f5d1db77f7a7bf4a7e"
server_key_version: "encryptionkey@0"
I20260926 15:09:31.531055 18556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/minicluster-data/master-0/wal/instance:
uuid: "f8c077ce1f004166989faaf5668ef1e8"
format_stamp: "Formatted at 2026-09-26 15:09:31 on dist-test-slave-865v"
server_key: "449fb37a37492bc0d57118fc48681ed2"
server_key_iv: "7e08b4583b7bf4f5d1db77f7a7bf4a7e"
server_key_version: "encryptionkey@0"
I20260926 15:09:31.536214 18556 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260926 15:09:31.539503 18571 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:09:31.540701 18556 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260926 15:09:31.540874 18556 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/minicluster-data/master-0/data,/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/minicluster-data/master-0/wal
uuid: "f8c077ce1f004166989faaf5668ef1e8"
format_stamp: "Formatted at 2026-09-26 15:09:31 on dist-test-slave-865v"
server_key: "449fb37a37492bc0d57118fc48681ed2"
server_key_iv: "7e08b4583b7bf4f5d1db77f7a7bf4a7e"
server_key_version: "encryptionkey@0"
I20260926 15:09:31.541055 18556 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:09:31.569511 18556 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:09:31.573424 18556 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:09:31.573693 18556 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:09:31.591594 18556 rpc_server.cc:307] RPC server started. Bound to: 127.12.34.254:42537
I20260926 15:09:31.591634 18622 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.34.254:42537 every 8 connection(s)
I20260926 15:09:31.592898 18556 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
I20260926 15:09:31.595973 12427 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8uoJGv/build/asan/bin/kudu as pid 18556
I20260926 15:09:31.596181 12427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/minicluster-data/master-0/wal/instance
I20260926 15:09:31.596791 18623 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:09:31.596844 12427 external_mini_cluster.cc:1428] Setting key 6eb599501d6301eaff5b32d6624234f8
I20260926 15:09:31.611488 18623 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8c077ce1f004166989faaf5668ef1e8: Bootstrap starting.
Sep 26 15:09:31 dist-test-slave-865v krb5kdc[18543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435371, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.12.34.254@KRBTEST.COM
I20260926 15:09:31.615026 18623 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f8c077ce1f004166989faaf5668ef1e8: Neither blocks nor log segments found. Creating new log.
I20260926 15:09:31.616197 18623 log.cc:824] T 00000000000000000000000000000000 P f8c077ce1f004166989faaf5668ef1e8: Log is configured to *not* fsync() on all Append() calls
I20260926 15:09:31.619282 18623 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8c077ce1f004166989faaf5668ef1e8: No bootstrap required, opened a new log
I20260926 15:09:31.620005 18626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:31.598771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36750 (local address 127.12.34.254:42537)
0926 15:09:31.599603 (+ 832us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:31.599624 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:31.599680 (+ 56us) server_negotiation.cc:407] Connection header received
0926 15:09:31.601889 (+ 2209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:31.601916 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:31.602432 (+ 516us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:31.602864 (+ 432us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:09:31.604147 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:31.606314 (+ 2167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:31.607515 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:31.608665 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:31.608866 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:31.612878 (+ 4012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:09:31.612886 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:09:31.612899 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:09:31.612936 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:09:31.615970 (+ 3034us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:31.616859 (+ 889us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:31.616871 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:31.616881 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:31.617117 (+ 236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:31.617527 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:31.617535 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:31.617539 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:31.618000 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:09:31.618209 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0926 15:09:31.618640 (+ 431us) server_negotiation.cc:299] Negotiation successful
0926 15:09:31.618966 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":595,"thread_start_us":317,"threads_started":1}
I20260926 15:09:31.626853 18623 raft_consensus.cc:348] T 00000000000000000000000000000000 P f8c077ce1f004166989faaf5668ef1e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8c077ce1f004166989faaf5668ef1e8" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 42537 } }
I20260926 15:09:31.627192 18623 raft_consensus.cc:374] T 00000000000000000000000000000000 P f8c077ce1f004166989faaf5668ef1e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:09:31.627319 18623 raft_consensus.cc:729] T 00000000000000000000000000000000 P f8c077ce1f004166989faaf5668ef1e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8c077ce1f004166989faaf5668ef1e8, State: Initialized, Role: FOLLOWER
I20260926 15:09:31.627826 18623 consensus_queue.cc:260] T 00000000000000000000000000000000 P f8c077ce1f004166989faaf5668ef1e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8c077ce1f004166989faaf5668ef1e8" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 42537 } }
I20260926 15:09:31.628028 18623 raft_consensus.cc:388] T 00000000000000000000000000000000 P f8c077ce1f004166989faaf5668ef1e8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 15:09:31.628152 18623 raft_consensus.cc:482] T 00000000000000000000000000000000 P f8c077ce1f004166989faaf5668ef1e8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 15:09:31.628311 18623 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f8c077ce1f004166989faaf5668ef1e8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:09:31.630656 18623 raft_consensus.cc:504] T 00000000000000000000000000000000 P f8c077ce1f004166989faaf5668ef1e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8c077ce1f004166989faaf5668ef1e8" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 42537 } }
I20260926 15:09:31.631304 18623 leader_election.cc:302] T 00000000000000000000000000000000 P f8c077ce1f004166989faaf5668ef1e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f8c077ce1f004166989faaf5668ef1e8; no voters:
I20260926 15:09:31.631886 18623 leader_election.cc:288] T 00000000000000000000000000000000 P f8c077ce1f004166989faaf5668ef1e8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 15:09:31.631982 18628 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f8c077ce1f004166989faaf5668ef1e8 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:09:31.632696 18628 raft_consensus.cc:686] T 00000000000000000000000000000000 P f8c077ce1f004166989faaf5668ef1e8 [term 1 LEADER]: Becoming Leader. State: Replica: f8c077ce1f004166989faaf5668ef1e8, State: Running, Role: LEADER
I20260926 15:09:31.633268 18628 consensus_queue.cc:237] T 00000000000000000000000000000000 P f8c077ce1f004166989faaf5668ef1e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8c077ce1f004166989faaf5668ef1e8" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 42537 } }
I20260926 15:09:31.633848 18623 sys_catalog.cc:564] T 00000000000000000000000000000000 P f8c077ce1f004166989faaf5668ef1e8 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:09:31.636037 18629 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8c077ce1f004166989faaf5668ef1e8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f8c077ce1f004166989faaf5668ef1e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8c077ce1f004166989faaf5668ef1e8" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 42537 } } }
I20260926 15:09:31.636016 18630 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8c077ce1f004166989faaf5668ef1e8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f8c077ce1f004166989faaf5668ef1e8. Latest consensus state: current_term: 1 leader_uuid: "f8c077ce1f004166989faaf5668ef1e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8c077ce1f004166989faaf5668ef1e8" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 42537 } } }
I20260926 15:09:31.636430 18629 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8c077ce1f004166989faaf5668ef1e8 [sys.catalog]: This master's current role is: LEADER
I20260926 15:09:31.636430 18630 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8c077ce1f004166989faaf5668ef1e8 [sys.catalog]: This master's current role is: LEADER
I20260926 15:09:31.638552 18634 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:09:31.643424 18634 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 15:09:31.653453 18634 catalog_manager.cc:1347] Generated new cluster ID: 165375e664a14f989e39db1da9eac0b9
I20260926 15:09:31.653929 18634 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 15:09:31.669675 18634 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 15:09:31.670920 18634 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:09:31.678234 18634 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f8c077ce1f004166989faaf5668ef1e8: Generated new TSK 0
I20260926 15:09:31.679051 18634 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.12.34.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.34.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.34.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/krb5kdc/oryx_127.12.34.193.keytab.
Entry for principal oryx/127.12.34.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/krb5kdc/oryx_127.12.34.193.keytab.
I20260926 15:09:31.730966 12427 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8uoJGv/build/asan/bin/kudu
/tmp/dist-test-task8uoJGv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.34.193:0
--local_ip_for_outbound_sockets=127.12.34.193
--webserver_interface=127.12.34.193
--webserver_port=0
--tserver_master_addrs=127.12.34.254:42537
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/krb5kdc/oryx_127.12.34.193.keytab
--principal=oryx/127.12.34.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.34.212:35299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:09:32.046921 18649 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:09:32.047160 18649 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:09:32.047329 18649 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:09:32.054266 18649 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:09:32.054366 18649 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:09:32.054582 18649 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.34.193
I20260926 15:09:32.063696 18649 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.34.212:35299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/krb5kdc/oryx_127.12.34.193.keytab
--principal=oryx/127.12.34.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.34.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.34.193
--webserver_port=0
--tserver_master_addrs=127.12.34.254:42537
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.34.193
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:09:32.064842 18649 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:09:32.066087 18649 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:09:32.080619 18657 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:09:32.080753 18654 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:09:32.080649 18655 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:09:32.081063 18649 server_base.cc:1034] running on GCE node
I20260926 15:09:32.081986 18649 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:09:32.083091 18649 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:09:32.084499 18649 hybrid_clock.cc:648] HybridClock initialized: now 1790435372084392 us; error 122 us; skew 500 ppm
I20260926 15:09:32.084887 18649 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:09:32 dist-test-slave-865v krb5kdc[18543](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435372, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.34.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:09:32.088196 18649 init.cc:377] Logged in from keytab as oryx/127.12.34.193@KRBTEST.COM (short username oryx)
I20260926 15:09:32.089706 18649 webserver.cc:460] Webserver started at http://127.12.34.193:40817/ using document root <none> and password file <none>
I20260926 15:09:32.090497 18649 fs_manager.cc:359] Metadata directory not provided
I20260926 15:09:32.090641 18649 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:09:32.090927 18649 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:09:32.094087 18649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/minicluster-data/ts-0/data/instance:
uuid: "d3e156f17705476c90ae1f60578daa75"
format_stamp: "Formatted at 2026-09-26 15:09:32 on dist-test-slave-865v"
server_key: "3b8749c227651f4b35758c4e969c6d5a"
server_key_iv: "10a9d62ce1a7f9ea3c3c316d6c550d8c"
server_key_version: "encryptionkey@0"
I20260926 15:09:32.094897 18649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/minicluster-data/ts-0/wal/instance:
uuid: "d3e156f17705476c90ae1f60578daa75"
format_stamp: "Formatted at 2026-09-26 15:09:32 on dist-test-slave-865v"
server_key: "3b8749c227651f4b35758c4e969c6d5a"
server_key_iv: "10a9d62ce1a7f9ea3c3c316d6c550d8c"
server_key_version: "encryptionkey@0"
I20260926 15:09:32.099746 18649 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260926 15:09:32.103129 18664 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:09:32.104493 18649 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 15:09:32.104656 18649 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/minicluster-data/ts-0/data,/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/minicluster-data/ts-0/wal
uuid: "d3e156f17705476c90ae1f60578daa75"
format_stamp: "Formatted at 2026-09-26 15:09:32 on dist-test-slave-865v"
server_key: "3b8749c227651f4b35758c4e969c6d5a"
server_key_iv: "10a9d62ce1a7f9ea3c3c316d6c550d8c"
server_key_version: "encryptionkey@0"
I20260926 15:09:32.104836 18649 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:09:32.124182 18649 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:09:32.128876 18649 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:09:32.129227 18649 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:09:32.130363 18649 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:09:32.131615 18649 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:09:32.131690 18649 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:09:32.131773 18649 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:09:32.131815 18649 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:09:32.163522 18649 rpc_server.cc:307] RPC server started. Bound to: 127.12.34.193:38979
I20260926 15:09:32.163555 18776 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.34.193:38979 every 8 connection(s)
I20260926 15:09:32.164781 18649 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/minicluster-data/ts-0/data/info.pb
I20260926 15:09:32.173731 12427 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8uoJGv/build/asan/bin/kudu as pid 18649
I20260926 15:09:32.173962 12427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/minicluster-data/ts-0/wal/instance
I20260926 15:09:32.174722 12427 external_mini_cluster.cc:1428] Setting key 11ad63e80d4f35611f5fa664bcb64770
Sep 26 15:09:32 dist-test-slave-865v krb5kdc[18543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435372, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.34.193@KRBTEST.COM for oryx/127.12.34.254@KRBTEST.COM
I20260926 15:09:32.186295 18779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:32.167315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.193:48973 (local address 127.12.34.254:42537)
0926 15:09:32.167786 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:32.167791 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:32.168548 (+ 757us) server_negotiation.cc:407] Connection header received
0926 15:09:32.170382 (+ 1834us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:32.170388 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:32.170478 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:32.170597 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:09:32.172012 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:32.172724 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:32.174940 (+ 2216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:32.175333 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:32.175478 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:32.179359 (+ 3881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:09:32.179368 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:09:32.179375 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:09:32.179418 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:09:32.182991 (+ 3573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:32.183919 (+ 928us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:32.183924 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:32.183927 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:32.184041 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:32.184582 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:32.184586 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:32.184588 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:32.184856 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:09:32.184976 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0926 15:09:32.185883 (+ 907us) server_negotiation.cc:299] Negotiation successful
0926 15:09:32.186052 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":273,"threads_started":1}
I20260926 15:09:32.187817 18780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:32.167683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:42537 (local address 127.12.34.193:48973)
0926 15:09:32.168371 (+ 688us) negotiation.cc:107] Waiting for socket to connect
0926 15:09:32.168416 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 15:09:32.170113 (+ 1697us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:09:32.170821 (+ 708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:09:32.170830 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:09:32.171320 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:09:32.171852 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:32.171867 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:32.172870 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:32.172880 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:32.174785 (+ 1905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:32.174801 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:32.175564 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:32.175573 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:32.175826 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:32.176566 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:09:32.176601 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:09:32.179159 (+ 2558us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:09:32.183139 (+ 3980us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:09:32.183146 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:09:32.183158 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:09:32.183787 (+ 629us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:09:32.184150 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:09:32.184156 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:09:32.184161 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:09:32.184399 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:09:32.184991 (+ 592us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:09:32.185002 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:09:32.185495 (+ 493us) client_negotiation.cc:769] Sending connection context
0926 15:09:32.185784 (+ 289us) client_negotiation.cc:240] Negotiation successful
0926 15:09:32.186315 (+ 531us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":282,"threads_started":1}
I20260926 15:09:32.189304 18777 heartbeater.cc:347] Connected to a master server at 127.12.34.254:42537
I20260926 15:09:32.189618 18777 heartbeater.cc:464] Registering TS with master...
I20260926 15:09:32.190340 18777 heartbeater.cc:511] Master 127.12.34.254:42537 requested a full tablet report, sending...
I20260926 15:09:32.192698 18588 ts_manager.cc:194] Registered new tserver with Master: d3e156f17705476c90ae1f60578daa75 (127.12.34.193:38979)
I20260926 15:09:32.194185 18588 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.12.34.193@KRBTEST.COM'} at 127.12.34.193:48973
WARNING: no policy specified for oryx/127.12.34.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.34.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.34.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/krb5kdc/oryx_127.12.34.194.keytab.
Entry for principal oryx/127.12.34.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/krb5kdc/oryx_127.12.34.194.keytab.
I20260926 15:09:32.223577 12427 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8uoJGv/build/asan/bin/kudu
/tmp/dist-test-task8uoJGv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.34.194:0
--local_ip_for_outbound_sockets=127.12.34.194
--webserver_interface=127.12.34.194
--webserver_port=0
--tserver_master_addrs=127.12.34.254:42537
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/krb5kdc/oryx_127.12.34.194.keytab
--principal=oryx/127.12.34.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.34.212:35299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:09:32.536972 18783 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:09:32.537254 18783 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:09:32.537344 18783 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:09:32.544603 18783 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:09:32.544699 18783 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:09:32.544816 18783 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.34.194
I20260926 15:09:32.554442 18783 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.34.212:35299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/krb5kdc/oryx_127.12.34.194.keytab
--principal=oryx/127.12.34.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.34.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.34.194
--webserver_port=0
--tserver_master_addrs=127.12.34.254:42537
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.34.194
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:09:32.555511 18783 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:09:32.556699 18783 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:09:32.575294 18788 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:09:32.575958 18791 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:09:32.576082 18789 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:09:32.576786 18783 server_base.cc:1034] running on GCE node
I20260926 15:09:32.577476 18783 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:09:32.578362 18783 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:09:32.579593 18783 hybrid_clock.cc:648] HybridClock initialized: now 1790435372579560 us; error 51 us; skew 500 ppm
I20260926 15:09:32.579895 18783 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:09:32 dist-test-slave-865v krb5kdc[18543](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435372, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.34.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:09:32.583050 18783 init.cc:377] Logged in from keytab as oryx/127.12.34.194@KRBTEST.COM (short username oryx)
I20260926 15:09:32.584391 18783 webserver.cc:460] Webserver started at http://127.12.34.194:46461/ using document root <none> and password file <none>
I20260926 15:09:32.585062 18783 fs_manager.cc:359] Metadata directory not provided
I20260926 15:09:32.585256 18783 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:09:32.585556 18783 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:09:32.588284 18783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/minicluster-data/ts-1/data/instance:
uuid: "85dd0480684b43909497b35c05e43908"
format_stamp: "Formatted at 2026-09-26 15:09:32 on dist-test-slave-865v"
server_key: "652b4e46885e8b018d96ded4a20a3aa5"
server_key_iv: "14038bfe6efb1c5f73193d3d114c5b2c"
server_key_version: "encryptionkey@0"
I20260926 15:09:32.589044 18783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/minicluster-data/ts-1/wal/instance:
uuid: "85dd0480684b43909497b35c05e43908"
format_stamp: "Formatted at 2026-09-26 15:09:32 on dist-test-slave-865v"
server_key: "652b4e46885e8b018d96ded4a20a3aa5"
server_key_iv: "14038bfe6efb1c5f73193d3d114c5b2c"
server_key_version: "encryptionkey@0"
I20260926 15:09:32.594722 18783 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 15:09:32.598721 18798 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:09:32.600349 18783 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 15:09:32.600497 18783 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/minicluster-data/ts-1/data,/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/minicluster-data/ts-1/wal
uuid: "85dd0480684b43909497b35c05e43908"
format_stamp: "Formatted at 2026-09-26 15:09:32 on dist-test-slave-865v"
server_key: "652b4e46885e8b018d96ded4a20a3aa5"
server_key_iv: "14038bfe6efb1c5f73193d3d114c5b2c"
server_key_version: "encryptionkey@0"
I20260926 15:09:32.600692 18783 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:09:32.621302 18783 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:09:32.625190 18783 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:09:32.625420 18783 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:09:32.626245 18783 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:09:32.627491 18783 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:09:32.627562 18783 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:09:32.627645 18783 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:09:32.627687 18783 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:09:32.662834 18783 rpc_server.cc:307] RPC server started. Bound to: 127.12.34.194:38821
I20260926 15:09:32.662920 18910 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.34.194:38821 every 8 connection(s)
I20260926 15:09:32.664072 18783 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/minicluster-data/ts-1/data/info.pb
I20260926 15:09:32.673276 12427 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8uoJGv/build/asan/bin/kudu as pid 18783
I20260926 15:09:32.673480 12427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/minicluster-data/ts-1/wal/instance
I20260926 15:09:32.673945 12427 external_mini_cluster.cc:1428] Setting key 4f01646ca274a12ba7bcf4fe8820108f
Sep 26 15:09:32 dist-test-slave-865v krb5kdc[18543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435372, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.34.194@KRBTEST.COM for oryx/127.12.34.254@KRBTEST.COM
I20260926 15:09:32.687168 18779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:32.666626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.194:53815 (local address 127.12.34.254:42537)
0926 15:09:32.666817 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:32.666821 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:32.668206 (+ 1385us) server_negotiation.cc:407] Connection header received
0926 15:09:32.669972 (+ 1766us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:32.669977 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:32.670052 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:32.670146 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:09:32.671798 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:32.672831 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:32.674822 (+ 1991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:32.675314 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:32.675408 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:32.679926 (+ 4518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:09:32.679936 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:09:32.679945 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:09:32.679997 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:09:32.683739 (+ 3742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:32.684769 (+ 1030us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:32.684774 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:32.684776 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:32.684917 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:32.685437 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:32.685441 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:32.685443 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:32.685707 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:09:32.685833 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0926 15:09:32.686712 (+ 879us) server_negotiation.cc:299] Negotiation successful
0926 15:09:32.686903 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 15:09:32.688657 18913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:32.667064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:42537 (local address 127.12.34.194:53815)
0926 15:09:32.667883 (+ 819us) negotiation.cc:107] Waiting for socket to connect
0926 15:09:32.667940 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0926 15:09:32.669678 (+ 1738us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:09:32.670452 (+ 774us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:09:32.670461 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:09:32.670990 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:09:32.671543 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:32.671559 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:32.673013 (+ 1454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:32.673017 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:32.674655 (+ 1638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:32.674670 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:32.675546 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:32.675553 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:32.675796 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:32.676650 (+ 854us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:09:32.676692 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:09:32.679680 (+ 2988us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:09:32.683895 (+ 4215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:09:32.683907 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:09:32.683927 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:09:32.684583 (+ 656us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:09:32.685036 (+ 453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:09:32.685044 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:09:32.685051 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:09:32.685310 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:09:32.685844 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:09:32.685855 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:09:32.686312 (+ 457us) client_negotiation.cc:769] Sending connection context
0926 15:09:32.686612 (+ 300us) client_negotiation.cc:240] Negotiation successful
0926 15:09:32.687200 (+ 588us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":334,"threads_started":1}
I20260926 15:09:32.690176 18911 heartbeater.cc:347] Connected to a master server at 127.12.34.254:42537
I20260926 15:09:32.690558 18911 heartbeater.cc:464] Registering TS with master...
I20260926 15:09:32.691329 18911 heartbeater.cc:511] Master 127.12.34.254:42537 requested a full tablet report, sending...
I20260926 15:09:32.692857 18588 ts_manager.cc:194] Registered new tserver with Master: 85dd0480684b43909497b35c05e43908 (127.12.34.194:38821)
I20260926 15:09:32.693830 18588 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.12.34.194@KRBTEST.COM'} at 127.12.34.194:53815
WARNING: no policy specified for oryx/127.12.34.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.34.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.34.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/krb5kdc/oryx_127.12.34.195.keytab.
Entry for principal oryx/127.12.34.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/krb5kdc/oryx_127.12.34.195.keytab.
I20260926 15:09:32.724148 12427 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8uoJGv/build/asan/bin/kudu
/tmp/dist-test-task8uoJGv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.34.195:0
--local_ip_for_outbound_sockets=127.12.34.195
--webserver_interface=127.12.34.195
--webserver_port=0
--tserver_master_addrs=127.12.34.254:42537
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/krb5kdc/oryx_127.12.34.195.keytab
--principal=oryx/127.12.34.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.34.212:35299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:09:33.040215 18916 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:09:33.040539 18916 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:09:33.040694 18916 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:09:33.048147 18916 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:09:33.048300 18916 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:09:33.048482 18916 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.34.195
I20260926 15:09:33.058712 18916 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.34.212:35299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/krb5kdc/oryx_127.12.34.195.keytab
--principal=oryx/127.12.34.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.34.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.34.195
--webserver_port=0
--tserver_master_addrs=127.12.34.254:42537
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.34.195
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:09:33.060070 18916 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:09:33.061488 18916 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:09:33.074680 18924 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:09:33.074708 18921 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:09:33.074890 18922 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:09:33.075647 18916 server_base.cc:1034] running on GCE node
I20260926 15:09:33.076236 18916 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:09:33.077272 18916 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:09:33.078474 18916 hybrid_clock.cc:648] HybridClock initialized: now 1790435373078417 us; error 53 us; skew 500 ppm
I20260926 15:09:33.078749 18916 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:09:33 dist-test-slave-865v krb5kdc[18543](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435373, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.34.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:09:33.081881 18916 init.cc:377] Logged in from keytab as oryx/127.12.34.195@KRBTEST.COM (short username oryx)
I20260926 15:09:33.083228 18916 webserver.cc:460] Webserver started at http://127.12.34.195:39429/ using document root <none> and password file <none>
I20260926 15:09:33.083859 18916 fs_manager.cc:359] Metadata directory not provided
I20260926 15:09:33.083956 18916 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:09:33.084183 18916 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:09:33.087127 18916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/minicluster-data/ts-2/data/instance:
uuid: "71e96bd442d34587b6d0e98bb1c8aae4"
format_stamp: "Formatted at 2026-09-26 15:09:33 on dist-test-slave-865v"
server_key: "7a2d78a764c744470291a7ee16def260"
server_key_iv: "39a23a6be6f0211c1a793bdfb27717c1"
server_key_version: "encryptionkey@0"
I20260926 15:09:33.087881 18916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/minicluster-data/ts-2/wal/instance:
uuid: "71e96bd442d34587b6d0e98bb1c8aae4"
format_stamp: "Formatted at 2026-09-26 15:09:33 on dist-test-slave-865v"
server_key: "7a2d78a764c744470291a7ee16def260"
server_key_iv: "39a23a6be6f0211c1a793bdfb27717c1"
server_key_version: "encryptionkey@0"
I20260926 15:09:33.092942 18916 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.006s
I20260926 15:09:33.096470 18931 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:09:33.098152 18916 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 15:09:33.098331 18916 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/minicluster-data/ts-2/data,/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/minicluster-data/ts-2/wal
uuid: "71e96bd442d34587b6d0e98bb1c8aae4"
format_stamp: "Formatted at 2026-09-26 15:09:33 on dist-test-slave-865v"
server_key: "7a2d78a764c744470291a7ee16def260"
server_key_iv: "39a23a6be6f0211c1a793bdfb27717c1"
server_key_version: "encryptionkey@0"
I20260926 15:09:33.098538 18916 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:09:33.114226 18916 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:09:33.118772 18916 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:09:33.119062 18916 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:09:33.119980 18916 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:09:33.121318 18916 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:09:33.121411 18916 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:09:33.121563 18916 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:09:33.121613 18916 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:09:33.155419 18916 rpc_server.cc:307] RPC server started. Bound to: 127.12.34.195:36967
I20260926 15:09:33.155457 19043 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.34.195:36967 every 8 connection(s)
I20260926 15:09:33.156615 18916 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/minicluster-data/ts-2/data/info.pb
I20260926 15:09:33.163272 12427 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8uoJGv/build/asan/bin/kudu as pid 18916
I20260926 15:09:33.163450 12427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/minicluster-data/ts-2/wal/instance
I20260926 15:09:33.164050 12427 external_mini_cluster.cc:1428] Setting key 5007528d4eed6e6d28bb8dc43cf4d84a
Sep 26 15:09:33 dist-test-slave-865v krb5kdc[18543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435373, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.34.195@KRBTEST.COM for oryx/127.12.34.254@KRBTEST.COM
I20260926 15:09:33.177379 18779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:33.159144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.195:44957 (local address 127.12.34.254:42537)
0926 15:09:33.159347 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:33.159351 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:33.160444 (+ 1093us) server_negotiation.cc:407] Connection header received
0926 15:09:33.162211 (+ 1767us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:33.162217 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:33.162318 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:33.162412 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:09:33.164589 (+ 2177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:33.165769 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:33.168059 (+ 2290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:33.168658 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:33.168791 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:33.172134 (+ 3343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:09:33.172138 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:09:33.172141 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:09:33.172171 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:09:33.174733 (+ 2562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:33.175454 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:33.175458 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:33.175461 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:33.175656 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:33.176023 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:33.176026 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:33.176028 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:33.176269 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:09:33.176374 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 15:09:33.176923 (+ 549us) server_negotiation.cc:299] Negotiation successful
0926 15:09:33.177072 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260926 15:09:33.178470 19046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:33.159606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:42537 (local address 127.12.34.195:44957)
0926 15:09:33.160286 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0926 15:09:33.160327 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 15:09:33.161950 (+ 1623us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:09:33.162632 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:09:33.162639 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:09:33.163359 (+ 720us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:09:33.164135 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:33.164161 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:33.165934 (+ 1773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:33.165939 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:33.167840 (+ 1901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:33.167877 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:33.168863 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:33.168869 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:33.169068 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:33.169859 (+ 791us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:09:33.169882 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:09:33.171952 (+ 2070us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:09:33.174868 (+ 2916us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:09:33.174874 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:09:33.174885 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:09:33.175315 (+ 430us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:09:33.175753 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:09:33.175757 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:09:33.175759 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:09:33.175931 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:09:33.176371 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:09:33.176376 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:09:33.176644 (+ 268us) client_negotiation.cc:769] Sending connection context
0926 15:09:33.176854 (+ 210us) client_negotiation.cc:240] Negotiation successful
0926 15:09:33.177260 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":258,"threads_started":1}
I20260926 15:09:33.179766 19044 heartbeater.cc:347] Connected to a master server at 127.12.34.254:42537
I20260926 15:09:33.180068 19044 heartbeater.cc:464] Registering TS with master...
I20260926 15:09:33.180727 19044 heartbeater.cc:511] Master 127.12.34.254:42537 requested a full tablet report, sending...
I20260926 15:09:33.182143 18587 ts_manager.cc:194] Registered new tserver with Master: 71e96bd442d34587b6d0e98bb1c8aae4 (127.12.34.195:36967)
I20260926 15:09:33.182922 18587 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.12.34.195@KRBTEST.COM'} at 127.12.34.195:44957
I20260926 15:09:33.192137 12427 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 15:09:33.196914 18777 heartbeater.cc:503] Master 127.12.34.254:42537 was elected leader, sending a full tablet report...
WARNING: no policy specified for oryx/127.12.34.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.34.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.34.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/krb5kdc/oryx_127.12.34.253.keytab.
Entry for principal oryx/127.12.34.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/krb5kdc/oryx_127.12.34.253.keytab.
I20260926 15:09:33.245048 12427 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8uoJGv/build/asan/bin/kudu
/tmp/dist-test-task8uoJGv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.34.253:32783
--webserver_interface=127.12.34.253
--webserver_port=0
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/krb5kdc/oryx_127.12.34.253.keytab
--principal=oryx/127.12.34.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.34.212:35299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.34.254:42537,127.12.34.253:32783
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:09:33.557463 19049 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:09:33.557735 19049 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:09:33.557832 19049 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:09:33.565066 19049 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:09:33.565266 19049 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:09:33.565346 19049 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:09:33.565433 19049 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:09:33.565500 19049 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:09:33.575517 19049 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.34.212:35299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.12.34.254:42537,127.12.34.253:32783
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/krb5kdc/oryx_127.12.34.253.keytab
--principal=oryx/127.12.34.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.34.253:32783
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.12.34.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
W20260926 15:09:33.575906 19049 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.12.34.254:42537,127.12.34.253:32783'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260926 15:09:33.576989 19049 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:09:33.578302 19049 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:09:33.589995 19054 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:09:33.589959 19055 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:09:33.589959 19057 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:09:33.590924 19049 server_base.cc:1034] running on GCE node
I20260926 15:09:33.591591 19049 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:09:33.593063 19049 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:09:33.596597 19049 hybrid_clock.cc:648] HybridClock initialized: now 1790435373596470 us; error 74 us; skew 500 ppm
I20260926 15:09:33.597464 19049 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:09:33 dist-test-slave-865v krb5kdc[18543](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435373, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.34.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:09:33.602149 19049 init.cc:377] Logged in from keytab as oryx/127.12.34.253@KRBTEST.COM (short username oryx)
I20260926 15:09:33.603914 19049 webserver.cc:460] Webserver started at http://127.12.34.253:35121/ using document root <none> and password file <none>
I20260926 15:09:33.604665 19049 fs_manager.cc:359] Metadata directory not provided
I20260926 15:09:33.604756 19049 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:09:33.605118 19049 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:09:33.608065 19049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/minicluster-data/master-1/data/instance:
uuid: "f183e2953d3141608ce600f9167cd229"
format_stamp: "Formatted at 2026-09-26 15:09:33 on dist-test-slave-865v"
server_key: "2908b928f18b5a51efd8eb48370b4be4"
server_key_iv: "2014418defd6e69d44792c03902ee391"
server_key_version: "encryptionkey@0"
I20260926 15:09:33.608825 19049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/minicluster-data/master-1/wal/instance:
uuid: "f183e2953d3141608ce600f9167cd229"
format_stamp: "Formatted at 2026-09-26 15:09:33 on dist-test-slave-865v"
server_key: "2908b928f18b5a51efd8eb48370b4be4"
server_key_iv: "2014418defd6e69d44792c03902ee391"
server_key_version: "encryptionkey@0"
I20260926 15:09:33.613934 19049 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260926 15:09:33.617401 19064 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:09:33.618906 19049 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 15:09:33.619082 19049 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/minicluster-data/master-1/data,/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/minicluster-data/master-1/wal
uuid: "f183e2953d3141608ce600f9167cd229"
format_stamp: "Formatted at 2026-09-26 15:09:33 on dist-test-slave-865v"
server_key: "2908b928f18b5a51efd8eb48370b4be4"
server_key_iv: "2014418defd6e69d44792c03902ee391"
server_key_version: "encryptionkey@0"
I20260926 15:09:33.619285 19049 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:09:33.649462 19049 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:09:33.653604 19049 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:09:33.653867 19049 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:09:33.672959 19049 rpc_server.cc:307] RPC server started. Bound to: 127.12.34.253:32783
I20260926 15:09:33.673043 19115 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.34.253:32783 every 8 connection(s)
I20260926 15:09:33.674244 19049 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/minicluster-data/master-1/data/info.pb
I20260926 15:09:33.678014 19116 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:09:33.684268 12427 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8uoJGv/build/asan/bin/kudu as pid 19049
I20260926 15:09:33.684305 19116 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.34.254" port: 42537 } has no permanent_uuid. Determining permanent_uuid...
I20260926 15:09:33.696467 18911 heartbeater.cc:503] Master 127.12.34.254:42537 was elected leader, sending a full tablet report...
Sep 26 15:09:33 dist-test-slave-865v krb5kdc[18543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435373, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.34.253@KRBTEST.COM for oryx/127.12.34.254@KRBTEST.COM
Sep 26 15:09:33 dist-test-slave-865v krb5kdc[18543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435371, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.12.34.253@KRBTEST.COM
I20260926 15:09:33.706176 19120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:33.686920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36754 (local address 127.12.34.254:42537)
0926 15:09:33.687438 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:33.687444 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:33.688059 (+ 615us) server_negotiation.cc:407] Connection header received
0926 15:09:33.689319 (+ 1260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:33.689324 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:33.689410 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:33.689513 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:09:33.691208 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:33.692449 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:33.694562 (+ 2113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:33.695186 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:33.695311 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:33.699102 (+ 3791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:09:33.699106 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:09:33.699110 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:09:33.699137 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:09:33.702676 (+ 3539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:33.703660 (+ 984us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:33.703666 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:33.703671 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:33.703822 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:33.704208 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:33.704213 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:33.704216 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:33.704640 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:09:33.704763 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0926 15:09:33.705652 (+ 889us) server_negotiation.cc:299] Negotiation successful
0926 15:09:33.705857 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":235,"threads_started":1}
I20260926 15:09:33.707187 19121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:33.687021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:42537 (local address 127.0.0.1:36754)
0926 15:09:33.687902 (+ 881us) negotiation.cc:107] Waiting for socket to connect
0926 15:09:33.687951 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 15:09:33.689003 (+ 1052us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:09:33.689593 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:09:33.689599 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:09:33.690462 (+ 863us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:09:33.691029 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:33.691053 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:33.692620 (+ 1567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:33.692624 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:33.694429 (+ 1805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:33.694443 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:33.695404 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:33.695408 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:33.695598 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:33.696339 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:09:33.696366 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:09:33.698923 (+ 2557us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:09:33.702817 (+ 3894us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:09:33.702825 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:09:33.702834 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:09:33.703523 (+ 689us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:09:33.703914 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:09:33.703921 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:09:33.703924 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:09:33.704094 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:09:33.704742 (+ 648us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:09:33.704750 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:09:33.705066 (+ 316us) client_negotiation.cc:769] Sending connection context
0926 15:09:33.705501 (+ 435us) client_negotiation.cc:240] Negotiation successful
0926 15:09:33.705977 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":627,"mutex_wait_us":47,"thread_start_us":478,"threads_started":1}
I20260926 15:09:33.707167 19119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:33.686279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44478 (local address 127.12.34.253:32783)
0926 15:09:33.687098 (+ 819us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:33.687117 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:33.687172 (+ 55us) server_negotiation.cc:407] Connection header received
0926 15:09:33.688975 (+ 1803us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:33.689000 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:33.689344 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:33.689549 (+ 205us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:09:33.691345 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:33.692915 (+ 1570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:33.694664 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:33.695485 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:33.695590 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:33.700145 (+ 4555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:09:33.700156 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:09:33.700172 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:09:33.700212 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:09:33.703742 (+ 3530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:33.704381 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:33.704389 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:33.704392 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:33.704518 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:33.704932 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:33.704941 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:33.704944 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:33.705350 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:09:33.705690 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0926 15:09:33.705975 (+ 285us) server_negotiation.cc:299] Negotiation successful
0926 15:09:33.706204 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":604,"thread_start_us":350,"threads_started":1}
I20260926 15:09:33.709177 19116 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.34.253" port: 32783 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 15:09:33 dist-test-slave-865v krb5kdc[18543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435373, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.34.253@KRBTEST.COM for oryx/127.12.34.253@KRBTEST.COM
I20260926 15:09:33.720862 19121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:33.710131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.253:32783 (local address 127.0.0.1:44480)
0926 15:09:33.710299 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0926 15:09:33.710314 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 15:09:33.710395 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:09:33.710594 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:09:33.710597 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:09:33.710862 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:09:33.710990 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:33.710996 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:33.711899 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:33.711903 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:33.712734 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:33.712743 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:33.713422 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:33.713426 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:33.713573 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:33.714022 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:09:33.714037 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:09:33.715852 (+ 1815us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:09:33.718766 (+ 2914us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:09:33.718770 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:09:33.718773 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:09:33.719154 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:09:33.719508 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:09:33.719511 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:09:33.719513 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:09:33.719631 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:09:33.720140 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:09:33.720144 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:09:33.720262 (+ 118us) client_negotiation.cc:769] Sending connection context
0926 15:09:33.720379 (+ 117us) client_negotiation.cc:240] Negotiation successful
0926 15:09:33.720587 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260926 15:09:33.720983 19119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:33.710128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44480 (local address 127.12.34.253:32783)
0926 15:09:33.710298 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:33.710303 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:33.710352 (+ 49us) server_negotiation.cc:407] Connection header received
0926 15:09:33.710462 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:33.710465 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:33.710512 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:33.710579 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:09:33.711099 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:33.711768 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:33.712848 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:33.713303 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:33.713402 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:33.716053 (+ 2651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:09:33.716057 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:09:33.716061 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:09:33.716086 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:09:33.718627 (+ 2541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:33.719289 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:33.719293 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:33.719295 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:33.719405 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:33.719749 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:33.719753 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:33.719755 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:33.720031 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:09:33.720140 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 15:09:33.720476 (+ 336us) server_negotiation.cc:299] Negotiation successful
0926 15:09:33.720698 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":53,"server-negotiator.queue_time_us":57}
I20260926 15:09:33.725841 19116 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f183e2953d3141608ce600f9167cd229: Bootstrap starting.
I20260926 15:09:33.728708 19116 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f183e2953d3141608ce600f9167cd229: Neither blocks nor log segments found. Creating new log.
I20260926 15:09:33.729730 19116 log.cc:824] T 00000000000000000000000000000000 P f183e2953d3141608ce600f9167cd229: Log is configured to *not* fsync() on all Append() calls
I20260926 15:09:33.732185 19116 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f183e2953d3141608ce600f9167cd229: No bootstrap required, opened a new log
I20260926 15:09:33.738946 19116 raft_consensus.cc:348] T 00000000000000000000000000000000 P f183e2953d3141608ce600f9167cd229 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c077ce1f004166989faaf5668ef1e8" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 42537 } } peers { permanent_uuid: "f183e2953d3141608ce600f9167cd229" member_type: VOTER last_known_addr { host: "127.12.34.253" port: 32783 } }
I20260926 15:09:33.739269 19116 raft_consensus.cc:374] T 00000000000000000000000000000000 P f183e2953d3141608ce600f9167cd229 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:09:33.739372 19116 raft_consensus.cc:729] T 00000000000000000000000000000000 P f183e2953d3141608ce600f9167cd229 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f183e2953d3141608ce600f9167cd229, State: Initialized, Role: FOLLOWER
I20260926 15:09:33.739974 19116 consensus_queue.cc:260] T 00000000000000000000000000000000 P f183e2953d3141608ce600f9167cd229 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c077ce1f004166989faaf5668ef1e8" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 42537 } } peers { permanent_uuid: "f183e2953d3141608ce600f9167cd229" member_type: VOTER last_known_addr { host: "127.12.34.253" port: 32783 } }
I20260926 15:09:33.740810 19123 sys_catalog.cc:455] T 00000000000000000000000000000000 P f183e2953d3141608ce600f9167cd229 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c077ce1f004166989faaf5668ef1e8" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 42537 } } peers { permanent_uuid: "f183e2953d3141608ce600f9167cd229" member_type: VOTER last_known_addr { host: "127.12.34.253" port: 32783 } } }
I20260926 15:09:33.741231 19123 sys_catalog.cc:458] T 00000000000000000000000000000000 P f183e2953d3141608ce600f9167cd229 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 15:09:33.741791 19116 sys_catalog.cc:564] T 00000000000000000000000000000000 P f183e2953d3141608ce600f9167cd229 [sys.catalog]: configured and running, proceeding with master startup.
W20260926 15:09:33.750329 19134 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f183e2953d3141608ce600f9167cd229: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 15:09:33.750469 19134 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 15:09:33.752624 19049 master_runner.cc:428] Detected that this master f183e2953d3141608ce600f9167cd229 is joining an existing cluster
I20260926 15:09:33.752722 19049 master_runner.cc:432] Initiating AddMaster RPC to add 127.12.34.253:32783
I20260926 15:09:33.777323 19119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:33.756760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44494 (local address 127.12.34.253:32783)
0926 15:09:33.756930 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:33.756936 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:33.757759 (+ 823us) server_negotiation.cc:407] Connection header received
0926 15:09:33.757994 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:33.757999 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:33.758070 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:33.758137 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:09:33.759614 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:33.760665 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:33.762667 (+ 2002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:33.763358 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:33.763458 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:33.770166 (+ 6708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:09:33.770173 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:09:33.770177 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:09:33.770210 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:09:33.774158 (+ 3948us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:33.775133 (+ 975us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:33.775139 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:33.775142 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:33.775298 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:33.775701 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:33.775708 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:33.775712 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:33.776096 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:09:33.776230 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0926 15:09:33.776694 (+ 464us) server_negotiation.cc:299] Negotiation successful
0926 15:09:33.776921 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 15:09:33.778187 19120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:33.756801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36768 (local address 127.12.34.254:42537)
0926 15:09:33.756970 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:33.756978 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:33.758240 (+ 1262us) server_negotiation.cc:407] Connection header received
0926 15:09:33.758420 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:33.758427 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:33.758503 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:33.758608 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:09:33.759624 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:33.760390 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:33.762452 (+ 2062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:33.763162 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:33.763275 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:33.770331 (+ 7056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:09:33.770350 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:09:33.770355 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:09:33.770400 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:09:33.774751 (+ 4351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:33.775518 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:33.775524 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:33.775527 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:33.775666 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:33.776081 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:33.776085 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:33.776088 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:33.776414 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:09:33.776544 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0926 15:09:33.777236 (+ 692us) server_negotiation.cc:299] Negotiation successful
0926 15:09:33.777861 (+ 625us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 15:09:33.778216 19140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:33.756808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.253:32783 (local address 127.0.0.1:44494)
0926 15:09:33.757659 (+ 851us) negotiation.cc:107] Waiting for socket to connect
0926 15:09:33.757679 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 15:09:33.757848 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:09:33.758304 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:09:33.758308 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:09:33.759026 (+ 718us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:09:33.759476 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:33.759488 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:33.760904 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:33.760912 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:33.762526 (+ 1614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:33.762541 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:33.763753 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:33.763759 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:33.763938 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:33.768578 (+ 4640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:09:33.768600 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:09:33.769993 (+ 1393us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:09:33.774325 (+ 4332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:09:33.774333 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:09:33.774339 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:09:33.775019 (+ 680us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:09:33.775426 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:09:33.775433 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:09:33.775439 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:09:33.775592 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:09:33.776294 (+ 702us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:09:33.776300 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:09:33.776452 (+ 152us) client_negotiation.cc:769] Sending connection context
0926 15:09:33.777516 (+ 1064us) client_negotiation.cc:240] Negotiation successful
0926 15:09:33.777794 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":712,"thread_start_us":409,"threads_started":1}
I20260926 15:09:33.779656 19141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:33.757256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:42537 (local address 127.0.0.1:36768)
0926 15:09:33.758142 (+ 886us) negotiation.cc:107] Waiting for socket to connect
0926 15:09:33.758155 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 15:09:33.758290 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:09:33.758627 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:09:33.758631 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:09:33.759199 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:09:33.759489 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:33.759498 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:33.760580 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:33.760585 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:33.762253 (+ 1668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:33.762268 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:33.763597 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:33.763606 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:33.763844 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:33.768533 (+ 4689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:09:33.768626 (+ 93us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:09:33.770100 (+ 1474us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:09:33.774907 (+ 4807us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:09:33.774912 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:09:33.774916 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:09:33.775394 (+ 478us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:09:33.775812 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:09:33.775816 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:09:33.775820 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:09:33.775948 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:09:33.776616 (+ 668us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:09:33.776620 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:09:33.776794 (+ 174us) client_negotiation.cc:769] Sending connection context
0926 15:09:33.778933 (+ 2139us) client_negotiation.cc:240] Negotiation successful
0926 15:09:33.779220 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":777,"spinlock_wait_cycles":24832,"thread_start_us":571,"threads_started":1}
WARNING: no policy specified for oryx/127.12.34.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.34.252@KRBTEST.COM" created.
Sep 26 15:09:33 dist-test-slave-865v krb5kdc[18543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435371, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.34.254@KRBTEST.COM for oryx/127.12.34.253@KRBTEST.COM
I20260926 15:09:33.801782 19119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:33.786920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44498 (local address 127.12.34.253:32783)
0926 15:09:33.787076 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:33.787082 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:33.787303 (+ 221us) server_negotiation.cc:407] Connection header received
0926 15:09:33.787583 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:33.787589 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:33.787689 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:33.787796 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:09:33.788965 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:33.790099 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:33.791947 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:33.792790 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:33.792928 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:33.796426 (+ 3498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:09:33.796431 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:09:33.796434 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:09:33.796460 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:09:33.799127 (+ 2667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:33.799836 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:33.799841 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:33.799845 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:33.799967 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:33.800338 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:33.800345 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:33.800348 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:33.800590 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:09:33.800743 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0926 15:09:33.801207 (+ 464us) server_negotiation.cc:299] Negotiation successful
0926 15:09:33.801413 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 15:09:33.801851 19143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:33.786587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.253:32783 (local address 127.0.0.1:44498)
0926 15:09:33.787178 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0926 15:09:33.787236 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0926 15:09:33.787394 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:09:33.787837 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:09:33.787842 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:09:33.788494 (+ 652us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:09:33.788802 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:33.788818 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:33.790255 (+ 1437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:33.790261 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:33.791796 (+ 1535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:33.791810 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:33.792960 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:33.792967 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:33.793249 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:33.793939 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:09:33.793970 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:09:33.796247 (+ 2277us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:09:33.799317 (+ 3070us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:09:33.799328 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:09:33.799335 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:09:33.799704 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:09:33.800072 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:09:33.800077 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:09:33.800081 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:09:33.800233 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:09:33.800743 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:09:33.800752 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:09:33.800912 (+ 160us) client_negotiation.cc:769] Sending connection context
0926 15:09:33.801121 (+ 209us) client_negotiation.cc:240] Negotiation successful
0926 15:09:33.801414 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":255,"threads_started":1}
I20260926 15:09:33.803859 18587 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.12.34.253:32783: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "f183e2953d3141608ce600f9167cd229"
member_type: NON_VOTER
last_known_addr {
host: "127.12.34.253"
port: 32783
}
attrs {
promote: true
}
}
dest_uuid: "f8c077ce1f004166989faaf5668ef1e8"
cas_config_opid_index: -1
I20260926 15:09:33.804831 18587 consensus_queue.cc:237] T 00000000000000000000000000000000 P f8c077ce1f004166989faaf5668ef1e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f8c077ce1f004166989faaf5668ef1e8" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 42537 } } peers { permanent_uuid: "f183e2953d3141608ce600f9167cd229" member_type: NON_VOTER last_known_addr { host: "127.12.34.253" port: 32783 } attrs { promote: true } }
I20260926 15:09:33.811110 19091 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f183e2953d3141608ce600f9167cd229 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:09:33.810258 19147 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f8c077ce1f004166989faaf5668ef1e8 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER f183e2953d3141608ce600f9167cd229 (127.12.34.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f8c077ce1f004166989faaf5668ef1e8" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 42537 } } peers { permanent_uuid: "f183e2953d3141608ce600f9167cd229" member_type: NON_VOTER last_known_addr { host: "127.12.34.253" port: 32783 } attrs { promote: true } } }
I20260926 15:09:33.814416 19091 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f183e2953d3141608ce600f9167cd229 [term 1 FOLLOWER]: Refusing update from remote peer f8c077ce1f004166989faaf5668ef1e8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260926 15:09:33.814774 19147 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.12.34.253:32783
I20260926 15:09:33.815521 19049 master.cc:537] Master@127.12.34.253:32783 shutting down...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.34.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/krb5kdc/oryx_127.12.34.252.keytab.
Entry for principal oryx/127.12.34.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/krb5kdc/oryx_127.12.34.252.keytab.
I20260926 15:09:33.816455 19146 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8c077ce1f004166989faaf5668ef1e8 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "f8c077ce1f004166989faaf5668ef1e8" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f8c077ce1f004166989faaf5668ef1e8" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 42537 } } peers { permanent_uuid: "f183e2953d3141608ce600f9167cd229" member_type: NON_VOTER last_known_addr { host: "127.12.34.253" port: 32783 } attrs { promote: true } } }
I20260926 15:09:33.816730 19146 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8c077ce1f004166989faaf5668ef1e8 [sys.catalog]: This master's current role is: LEADER
I20260926 15:09:33.817453 12427 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8uoJGv/build/asan/bin/kudu
/tmp/dist-test-task8uoJGv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.34.252:44557
--webserver_interface=127.12.34.252
--webserver_port=0
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/krb5kdc/oryx_127.12.34.252.keytab
--principal=oryx/127.12.34.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.34.212:35299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.34.254:42537,127.12.34.253:32783,127.12.34.252:44557
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 15:09:33.821405 19146 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f8c077ce1f004166989faaf5668ef1e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f183e2953d3141608ce600f9167cd229" member_type: NON_VOTER last_known_addr { host: "127.12.34.253" port: 32783 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:09:33.828012 19049 raft_consensus.cc:2229] T 00000000000000000000000000000000 P f183e2953d3141608ce600f9167cd229 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 15:09:33.828416 19049 raft_consensus.cc:2258] T 00000000000000000000000000000000 P f183e2953d3141608ce600f9167cd229 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:09:33.828603 19049 tablet_replica.cc:330] T 00000000000000000000000000000000 P f183e2953d3141608ce600f9167cd229: stopping tablet replica
W20260926 15:09:33.829366 18575 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260926 15:09:33.835553 19049 master.cc:552] Master@127.12.34.253:32783 shutdown complete.
I20260926 15:09:33.835626 19049 master_runner.cc:305] Clearing existing system tablet
I20260926 15:09:33.836016 19049 fs_manager.cc:359] Metadata directory not provided
I20260926 15:09:33.836102 19049 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
W20260926 15:09:33.838594 18575 consensus_peers.cc:487] T 00000000000000000000000000000000 P f8c077ce1f004166989faaf5668ef1e8 -> Peer f183e2953d3141608ce600f9167cd229 (127.12.34.253:32783): Couldn't send request to peer f183e2953d3141608ce600f9167cd229. Status: Network error: recv got EOF from unknown peer (error 108). This is attempt 1: this message will repeat every 5th retry.
I20260926 15:09:33.839444 19049 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260926 15:09:33.841179 19153 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:09:33.841480 19049 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:09:33.841741 19049 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 15:09:33.841825 19049 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/minicluster-data/master-1/data,/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/minicluster-data/master-1/wal
uuid: "f183e2953d3141608ce600f9167cd229"
format_stamp: "Formatted at 2026-09-26 15:09:33 on dist-test-slave-865v"
server_key: "2908b928f18b5a51efd8eb48370b4be4"
server_key_iv: "2014418defd6e69d44792c03902ee391"
server_key_version: "encryptionkey@0"
I20260926 15:09:33.846765 19049 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P f183e2953d3141608ce600f9167cd229: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:09:33.853430 19049 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P f183e2953d3141608ce600f9167cd229: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260926 15:09:33.853588 19049 log.cc:1196] T 00000000000000000000000000000000 P f183e2953d3141608ce600f9167cd229: Deleting WAL directory at /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260926 15:09:33.853988 19049 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P f183e2953d3141608ce600f9167cd229: Deleting consensus metadata
I20260926 15:09:33.854254 19049 master_runner.cc:315] Copying system tablet from 127.12.34.254:42537
I20260926 15:09:33.858479 19049 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P f183e2953d3141608ce600f9167cd229: tablet copy: Beginning tablet copy session from remote peer at address 127.12.34.254:42537
I20260926 15:09:33.873490 19120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:33.859205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36776 (local address 127.12.34.254:42537)
0926 15:09:33.859405 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:33.859412 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:33.859873 (+ 461us) server_negotiation.cc:407] Connection header received
0926 15:09:33.860035 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:33.860040 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:33.860139 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:33.860247 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:09:33.861228 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:33.862320 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:33.864130 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:33.864719 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:33.864842 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:33.866990 (+ 2148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:09:33.866996 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:09:33.867000 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:09:33.867037 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:09:33.870594 (+ 3557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:33.871327 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:33.871349 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:33.871354 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:33.871537 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:33.871951 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:33.871956 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:33.871961 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:33.872305 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:09:33.872469 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0926 15:09:33.872878 (+ 409us) server_negotiation.cc:299] Negotiation successful
0926 15:09:33.873130 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260926 15:09:33.873533 19159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:33.859152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:42537 (local address 127.0.0.1:36776)
0926 15:09:33.859755 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0926 15:09:33.859774 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 15:09:33.859913 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:09:33.860293 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:09:33.860298 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:09:33.860794 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:09:33.861053 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:33.861064 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:33.862485 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:33.862491 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:33.863982 (+ 1491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:33.863996 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:33.864876 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:33.864882 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:33.865063 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:33.865773 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:09:33.865796 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:09:33.866769 (+ 973us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:09:33.870724 (+ 3955us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:09:33.870730 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:09:33.870734 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:09:33.871184 (+ 450us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:09:33.871671 (+ 487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:09:33.871679 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:09:33.871682 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:09:33.871833 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:09:33.872441 (+ 608us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:09:33.872447 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:09:33.872612 (+ 165us) client_negotiation.cc:769] Sending connection context
0926 15:09:33.872759 (+ 147us) client_negotiation.cc:240] Negotiation successful
0926 15:09:33.873039 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":352,"threads_started":1}
I20260926 15:09:33.874653 18608 tablet_copy_service.cc:140] P f8c077ce1f004166989faaf5668ef1e8: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer f183e2953d3141608ce600f9167cd229 ({username='oryx', principal='oryx/127.12.34.253@KRBTEST.COM'} at 127.0.0.1:36776)
I20260926 15:09:33.874956 18608 tablet_copy_service.cc:161] P f8c077ce1f004166989faaf5668ef1e8: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer f183e2953d3141608ce600f9167cd229 at {username='oryx', principal='oryx/127.12.34.253@KRBTEST.COM'} at 127.0.0.1:36776: session id = f183e2953d3141608ce600f9167cd229-00000000000000000000000000000000
I20260926 15:09:33.880179 18608 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P f8c077ce1f004166989faaf5668ef1e8: Tablet Copy: opened 0 blocks and 1 log segments
I20260926 15:09:33.882421 19049 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260926 15:09:33.889382 19049 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P f183e2953d3141608ce600f9167cd229: tablet copy: Starting download of 0 data blocks...
I20260926 15:09:33.889678 19049 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P f183e2953d3141608ce600f9167cd229: tablet copy: Starting download of 1 WAL segments...
I20260926 15:09:33.892361 19049 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P f183e2953d3141608ce600f9167cd229: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260926 15:09:33.896613 18608 tablet_copy_service.cc:342] P f8c077ce1f004166989faaf5668ef1e8: Request end of tablet copy session f183e2953d3141608ce600f9167cd229-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.12.34.253@KRBTEST.COM'} at 127.0.0.1:36776
I20260926 15:09:33.896808 18608 tablet_copy_service.cc:433] P f8c077ce1f004166989faaf5668ef1e8: ending tablet copy session f183e2953d3141608ce600f9167cd229-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer f183e2953d3141608ce600f9167cd229
I20260926 15:09:33.908862 19049 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:09:33.911537 19160 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:09:33.914448 19161 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:09:33.927012 19163 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:09:33.930482 19049 server_base.cc:1034] running on GCE node
I20260926 15:09:33.930853 19049 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:09:33.931515 19049 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:09:33.932765 19049 hybrid_clock.cc:648] HybridClock initialized: now 1790435373932690 us; error 38 us; skew 500 ppm
I20260926 15:09:33.933010 19049 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:09:33 dist-test-slave-865v krb5kdc[18543](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435373, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.34.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:09:33.935257 19049 init.cc:377] Logged in from keytab as oryx/127.12.34.253@KRBTEST.COM (short username oryx)
I20260926 15:09:33.940217 19049 webserver.cc:460] Webserver started at http://127.12.34.253:33595/ using document root <none> and password file <none>
I20260926 15:09:33.940614 19049 fs_manager.cc:359] Metadata directory not provided
I20260926 15:09:33.940694 19049 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:09:33.943861 19049 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260926 15:09:33.945886 19170 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:09:33.946307 19049 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 15:09:33.946455 19049 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/minicluster-data/master-1/data,/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/minicluster-data/master-1/wal
uuid: "f183e2953d3141608ce600f9167cd229"
format_stamp: "Formatted at 2026-09-26 15:09:33 on dist-test-slave-865v"
server_key: "2908b928f18b5a51efd8eb48370b4be4"
server_key_iv: "2014418defd6e69d44792c03902ee391"
server_key_version: "encryptionkey@0"
I20260926 15:09:33.946632 19049 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:09:33.962476 19049 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:09:33.967196 19049 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:09:34.000470 19049 rpc_server.cc:307] RPC server started. Bound to: 127.12.34.253:32783
I20260926 15:09:34.000492 19221 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.34.253:32783 every 8 connection(s)
I20260926 15:09:34.001387 19049 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/minicluster-data/master-1/data/info.pb
I20260926 15:09:34.008683 19222 sys_catalog.cc:263] Verifying existing consensus state
I20260926 15:09:34.012398 19222 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f183e2953d3141608ce600f9167cd229: Bootstrap starting.
I20260926 15:09:34.024303 19222 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f183e2953d3141608ce600f9167cd229: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 15:09:34.024755 19222 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f183e2953d3141608ce600f9167cd229: Bootstrap complete.
I20260926 15:09:34.025642 19222 raft_consensus.cc:348] T 00000000000000000000000000000000 P f183e2953d3141608ce600f9167cd229 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f8c077ce1f004166989faaf5668ef1e8" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 42537 } } peers { permanent_uuid: "f183e2953d3141608ce600f9167cd229" member_type: NON_VOTER last_known_addr { host: "127.12.34.253" port: 32783 } attrs { promote: true } }
I20260926 15:09:34.025868 19222 raft_consensus.cc:729] T 00000000000000000000000000000000 P f183e2953d3141608ce600f9167cd229 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f183e2953d3141608ce600f9167cd229, State: Initialized, Role: LEARNER
I20260926 15:09:34.026093 19222 consensus_queue.cc:260] T 00000000000000000000000000000000 P f183e2953d3141608ce600f9167cd229 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f8c077ce1f004166989faaf5668ef1e8" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 42537 } } peers { permanent_uuid: "f183e2953d3141608ce600f9167cd229" member_type: NON_VOTER last_known_addr { host: "127.12.34.253" port: 32783 } attrs { promote: true } }
I20260926 15:09:34.027148 19225 sys_catalog.cc:455] T 00000000000000000000000000000000 P f183e2953d3141608ce600f9167cd229 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f8c077ce1f004166989faaf5668ef1e8" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 42537 } } peers { permanent_uuid: "f183e2953d3141608ce600f9167cd229" member_type: NON_VOTER last_known_addr { host: "127.12.34.253" port: 32783 } attrs { promote: true } } }
I20260926 15:09:34.027506 19222 sys_catalog.cc:564] T 00000000000000000000000000000000 P f183e2953d3141608ce600f9167cd229 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:09:34.027536 19225 sys_catalog.cc:458] T 00000000000000000000000000000000 P f183e2953d3141608ce600f9167cd229 [sys.catalog]: This master's current role is: LEARNER
I20260926 15:09:34.035923 19236 catalog_manager.cc:1259] Loaded cluster ID: 165375e664a14f989e39db1da9eac0b9
I20260926 15:09:34.036031 19236 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f183e2953d3141608ce600f9167cd229: loading cluster ID for follower catalog manager: success
I20260926 15:09:34.038357 19236 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f183e2953d3141608ce600f9167cd229: acquiring CA information for follower catalog manager: success
I20260926 15:09:34.040205 19236 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f183e2953d3141608ce600f9167cd229: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 15:09:34.185269 19044 heartbeater.cc:503] Master 127.12.34.254:42537 was elected leader, sending a full tablet report...
W20260926 15:09:34.211758 19150 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:09:34.212018 19150 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:09:34.212155 19150 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:09:34.220038 19150 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:09:34.220139 19150 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:09:34.220216 19150 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:09:34.220311 19150 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:09:34.220394 19150 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:09:34.232833 19150 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.34.212:35299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.12.34.254:42537,127.12.34.253:32783,127.12.34.252:44557
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/krb5kdc/oryx_127.12.34.252.keytab
--principal=oryx/127.12.34.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.34.252:44557
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.12.34.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:09:34.233376 19143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:34.224773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.253:32783 (local address 127.0.0.1:44514)
0926 15:09:34.225024 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0926 15:09:34.225044 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 15:09:34.225196 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:09:34.226113 (+ 917us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:09:34.226118 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:09:34.226150 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:09:34.226342 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:34.226355 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:34.228871 (+ 2516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:34.228877 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:34.230844 (+ 1967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:34.230857 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:34.232520 (+ 1663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:34.232529 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:34.232743 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:34.232816 (+ 73us) client_negotiation.cc:769] Sending connection context
0926 15:09:34.232915 (+ 99us) client_negotiation.cc:240] Negotiation successful
0926 15:09:34.233055 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134}
I20260926 15:09:34.233657 19240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:34.224986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44514 (local address 127.12.34.253:32783)
0926 15:09:34.225726 (+ 740us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:34.225733 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:34.225752 (+ 19us) server_negotiation.cc:407] Connection header received
0926 15:09:34.225821 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:34.225826 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:34.225925 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:34.226047 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:09:34.226523 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:34.228714 (+ 2191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:34.231017 (+ 2303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:34.232333 (+ 1316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:34.232476 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:34.232734 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0926 15:09:34.232975 (+ 241us) server_negotiation.cc:299] Negotiation successful
0926 15:09:34.233144 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":601,"thread_start_us":438,"threads_started":1}
I20260926 15:09:34.234545 19150 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:09:34.234750 19197 raft_consensus.cc:1203] T 00000000000000000000000000000000 P f183e2953d3141608ce600f9167cd229 [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260926 15:09:34.235359 19225 sys_catalog.cc:455] T 00000000000000000000000000000000 P f183e2953d3141608ce600f9167cd229 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f8c077ce1f004166989faaf5668ef1e8. Latest consensus state: current_term: 1 leader_uuid: "f8c077ce1f004166989faaf5668ef1e8" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f8c077ce1f004166989faaf5668ef1e8" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 42537 } } peers { permanent_uuid: "f183e2953d3141608ce600f9167cd229" member_type: NON_VOTER last_known_addr { host: "127.12.34.253" port: 32783 } attrs { promote: true } } }
I20260926 15:09:34.235626 19225 sys_catalog.cc:458] T 00000000000000000000000000000000 P f183e2953d3141608ce600f9167cd229 [sys.catalog]: This master's current role is: LEARNER
I20260926 15:09:34.236097 19150 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:09:34.248502 19243 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:09:34.248569 19245 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:09:34.249146 19242 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:09:34.249559 19150 server_base.cc:1034] running on GCE node
I20260926 15:09:34.250401 19150 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:09:34.251984 19150 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:09:34.253172 19150 hybrid_clock.cc:648] HybridClock initialized: now 1790435374253098 us; error 67 us; skew 500 ppm
I20260926 15:09:34.253539 19150 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:09:34 dist-test-slave-865v krb5kdc[18543](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435374, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.34.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:09:34.257205 19150 init.cc:377] Logged in from keytab as oryx/127.12.34.252@KRBTEST.COM (short username oryx)
I20260926 15:09:34.258548 19150 webserver.cc:460] Webserver started at http://127.12.34.252:44397/ using document root <none> and password file <none>
I20260926 15:09:34.259217 19150 fs_manager.cc:359] Metadata directory not provided
I20260926 15:09:34.259337 19150 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:09:34.259634 19150 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:09:34.262509 19150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/minicluster-data/master-2/data/instance:
uuid: "70d9c6c1008544b4b7ba623a53d9e680"
format_stamp: "Formatted at 2026-09-26 15:09:34 on dist-test-slave-865v"
server_key: "2ef40050b8fc7f4a02a491d85060f093"
server_key_iv: "89d3fc1a788098cfba0e00eb187c0532"
server_key_version: "encryptionkey@0"
I20260926 15:09:34.263252 19150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/minicluster-data/master-2/wal/instance:
uuid: "70d9c6c1008544b4b7ba623a53d9e680"
format_stamp: "Formatted at 2026-09-26 15:09:34 on dist-test-slave-865v"
server_key: "2ef40050b8fc7f4a02a491d85060f093"
server_key_iv: "89d3fc1a788098cfba0e00eb187c0532"
server_key_version: "encryptionkey@0"
I20260926 15:09:34.268005 19150 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260926 15:09:34.271186 19252 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:09:34.272351 19150 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 15:09:34.272521 19150 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/minicluster-data/master-2/data,/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/minicluster-data/master-2/wal
uuid: "70d9c6c1008544b4b7ba623a53d9e680"
format_stamp: "Formatted at 2026-09-26 15:09:34 on dist-test-slave-865v"
server_key: "2ef40050b8fc7f4a02a491d85060f093"
server_key_iv: "89d3fc1a788098cfba0e00eb187c0532"
server_key_version: "encryptionkey@0"
I20260926 15:09:34.272691 19150 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:09:34.286983 19150 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:09:34.291468 19150 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:09:34.291744 19150 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:09:34.310076 19150 rpc_server.cc:307] RPC server started. Bound to: 127.12.34.252:44557
I20260926 15:09:34.310108 19303 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.34.252:44557 every 8 connection(s)
I20260926 15:09:34.311609 19150 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/minicluster-data/master-2/data/info.pb
I20260926 15:09:34.315137 19304 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:09:34.315938 12427 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8uoJGv/build/asan/bin/kudu as pid 19150
I20260926 15:09:34.323011 19304 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.34.254" port: 42537 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 15:09:34 dist-test-slave-865v krb5kdc[18543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435371, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.12.34.252@KRBTEST.COM
Sep 26 15:09:34 dist-test-slave-865v krb5kdc[18543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435374, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.34.252@KRBTEST.COM for oryx/127.12.34.254@KRBTEST.COM
I20260926 15:09:34.340431 19307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:34.317860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54080 (local address 127.12.34.252:44557)
0926 15:09:34.318617 (+ 757us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:34.318628 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:34.318666 (+ 38us) server_negotiation.cc:407] Connection header received
0926 15:09:34.320630 (+ 1964us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:34.320650 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:34.321000 (+ 350us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:34.321447 (+ 447us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:09:34.322773 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:34.324468 (+ 1695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:34.325881 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:34.326893 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:34.327048 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:34.331723 (+ 4675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:09:34.331733 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:09:34.331746 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:09:34.331790 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:09:34.335353 (+ 3563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:34.336269 (+ 916us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:34.336277 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:34.336285 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:34.336460 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:34.336971 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:34.336979 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:34.336985 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:34.337766 (+ 781us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:09:34.338073 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0926 15:09:34.338598 (+ 525us) server_negotiation.cc:299] Negotiation successful
0926 15:09:34.339060 (+ 462us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":570,"thread_start_us":337,"threads_started":1}
I20260926 15:09:34.342638 19120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:34.325739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36778 (local address 127.12.34.254:42537)
0926 15:09:34.325938 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:34.325947 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:34.326556 (+ 609us) server_negotiation.cc:407] Connection header received
0926 15:09:34.326702 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:34.326706 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:34.326773 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:34.326852 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:09:34.328005 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:34.329170 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:34.330801 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:34.331438 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:34.331567 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:34.335577 (+ 4010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:09:34.335582 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:09:34.335586 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:09:34.335617 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:09:34.339634 (+ 4017us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:34.340439 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:34.340446 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:34.340452 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:34.340646 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:34.341133 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:34.341138 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:34.341147 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:34.341416 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:09:34.341524 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 15:09:34.342040 (+ 516us) server_negotiation.cc:299] Negotiation successful
0926 15:09:34.342250 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 15:09:34.342691 19308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:34.325620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:42537 (local address 127.0.0.1:36778)
0926 15:09:34.326401 (+ 781us) negotiation.cc:107] Waiting for socket to connect
0926 15:09:34.326446 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 15:09:34.326593 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:09:34.326946 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:09:34.326955 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:09:34.327610 (+ 655us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:09:34.327830 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:34.327838 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:34.329294 (+ 1456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:34.329297 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:34.330616 (+ 1319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:34.330628 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:34.331652 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:34.331661 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:34.331863 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:34.332496 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:09:34.332524 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:09:34.335390 (+ 2866us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:09:34.339815 (+ 4425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:09:34.339823 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:09:34.339828 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:09:34.340317 (+ 489us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:09:34.340767 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:09:34.340772 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:09:34.340779 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:09:34.340980 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:09:34.341541 (+ 561us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:09:34.341551 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:09:34.341730 (+ 179us) client_negotiation.cc:769] Sending connection context
0926 15:09:34.341967 (+ 237us) client_negotiation.cc:240] Negotiation successful
0926 15:09:34.342259 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":406,"threads_started":1}
I20260926 15:09:34.344063 19304 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.34.253" port: 32783 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 15:09:34 dist-test-slave-865v krb5kdc[18543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435374, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.34.252@KRBTEST.COM for oryx/127.12.34.253@KRBTEST.COM
I20260926 15:09:34.356199 19308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:34.344823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.253:32783 (local address 127.0.0.1:44516)
0926 15:09:34.345023 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0926 15:09:34.345040 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 15:09:34.345154 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:09:34.345469 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:09:34.345473 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:09:34.345802 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:09:34.345917 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:34.345925 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:34.346875 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:34.346880 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:34.347898 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:34.347909 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:34.348676 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:34.348682 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:34.348855 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:34.349393 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:09:34.349408 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:09:34.351287 (+ 1879us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:09:34.354155 (+ 2868us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:09:34.354159 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:09:34.354161 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:09:34.354523 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:09:34.354899 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:09:34.354902 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:09:34.354904 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:09:34.355077 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:09:34.355590 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:09:34.355593 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:09:34.355700 (+ 107us) client_negotiation.cc:769] Sending connection context
0926 15:09:34.355803 (+ 103us) client_negotiation.cc:240] Negotiation successful
0926 15:09:34.355990 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260926 15:09:34.356678 19240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:34.344895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44516 (local address 127.12.34.253:32783)
0926 15:09:34.345106 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:34.345113 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:34.345131 (+ 18us) server_negotiation.cc:407] Connection header received
0926 15:09:34.345293 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:34.345297 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:34.345354 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:34.345480 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:09:34.346058 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:34.346739 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:34.348048 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:34.348530 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:34.348670 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:34.351466 (+ 2796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:09:34.351470 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:09:34.351474 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:09:34.351503 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:09:34.354043 (+ 2540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:34.354654 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:34.354658 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:34.354660 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:34.354798 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:34.355202 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:34.355206 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:34.355208 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:34.355491 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:09:34.355600 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 15:09:34.355917 (+ 317us) server_negotiation.cc:299] Negotiation successful
0926 15:09:34.356095 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 15:09:34.357697 19304 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.34.252" port: 44557 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 15:09:34 dist-test-slave-865v krb5kdc[18543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435374, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.34.252@KRBTEST.COM for oryx/127.12.34.252@KRBTEST.COM
I20260926 15:09:34.370376 19308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:34.358297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.252:44557 (local address 127.0.0.1:54082)
0926 15:09:34.358451 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 15:09:34.358465 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 15:09:34.358536 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:09:34.358797 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:09:34.358800 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:09:34.359104 (+ 304us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:09:34.359224 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:34.359231 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:34.360114 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:34.360118 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:34.360949 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:34.360960 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:34.361690 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:34.361694 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:34.362230 (+ 536us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:34.362773 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:09:34.362788 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:09:34.364675 (+ 1887us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:09:34.367961 (+ 3286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:09:34.367965 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:09:34.367967 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:09:34.368307 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:09:34.368674 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:09:34.368677 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:09:34.368680 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:09:34.368828 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:09:34.369464 (+ 636us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:09:34.369467 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:09:34.369605 (+ 138us) client_negotiation.cc:769] Sending connection context
0926 15:09:34.369738 (+ 133us) client_negotiation.cc:240] Negotiation successful
0926 15:09:34.370053 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260926 15:09:34.370597 19307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:34.358447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54082 (local address 127.12.34.252:44557)
0926 15:09:34.358583 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:34.358587 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:34.358600 (+ 13us) server_negotiation.cc:407] Connection header received
0926 15:09:34.358634 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:34.358637 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:34.358683 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:34.358753 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:09:34.359333 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:34.359991 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:34.361118 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:34.361528 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:34.361624 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:34.364852 (+ 3228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:09:34.364857 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:09:34.364867 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:09:34.364893 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:09:34.367812 (+ 2919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:34.368429 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:34.368433 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:34.368435 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:34.368549 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:34.368976 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:34.368980 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:34.368982 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:34.369323 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:09:34.369442 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0926 15:09:34.369820 (+ 378us) server_negotiation.cc:299] Negotiation successful
0926 15:09:34.370219 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":203,"server-negotiator.queue_time_us":49}
I20260926 15:09:34.375108 19304 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70d9c6c1008544b4b7ba623a53d9e680: Bootstrap starting.
I20260926 15:09:34.378240 19304 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 70d9c6c1008544b4b7ba623a53d9e680: Neither blocks nor log segments found. Creating new log.
I20260926 15:09:34.379185 19304 log.cc:824] T 00000000000000000000000000000000 P 70d9c6c1008544b4b7ba623a53d9e680: Log is configured to *not* fsync() on all Append() calls
I20260926 15:09:34.381698 19304 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70d9c6c1008544b4b7ba623a53d9e680: No bootstrap required, opened a new log
I20260926 15:09:34.388813 19304 raft_consensus.cc:348] T 00000000000000000000000000000000 P 70d9c6c1008544b4b7ba623a53d9e680 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c077ce1f004166989faaf5668ef1e8" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 42537 } } peers { permanent_uuid: "f183e2953d3141608ce600f9167cd229" member_type: VOTER last_known_addr { host: "127.12.34.253" port: 32783 } } peers { permanent_uuid: "70d9c6c1008544b4b7ba623a53d9e680" member_type: VOTER last_known_addr { host: "127.12.34.252" port: 44557 } }
I20260926 15:09:34.389146 19304 raft_consensus.cc:374] T 00000000000000000000000000000000 P 70d9c6c1008544b4b7ba623a53d9e680 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:09:34.389281 19304 raft_consensus.cc:729] T 00000000000000000000000000000000 P 70d9c6c1008544b4b7ba623a53d9e680 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70d9c6c1008544b4b7ba623a53d9e680, State: Initialized, Role: FOLLOWER
I20260926 15:09:34.389798 19304 consensus_queue.cc:260] T 00000000000000000000000000000000 P 70d9c6c1008544b4b7ba623a53d9e680 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c077ce1f004166989faaf5668ef1e8" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 42537 } } peers { permanent_uuid: "f183e2953d3141608ce600f9167cd229" member_type: VOTER last_known_addr { host: "127.12.34.253" port: 32783 } } peers { permanent_uuid: "70d9c6c1008544b4b7ba623a53d9e680" member_type: VOTER last_known_addr { host: "127.12.34.252" port: 44557 } }
I20260926 15:09:34.390560 19310 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70d9c6c1008544b4b7ba623a53d9e680 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8c077ce1f004166989faaf5668ef1e8" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 42537 } } peers { permanent_uuid: "f183e2953d3141608ce600f9167cd229" member_type: VOTER last_known_addr { host: "127.12.34.253" port: 32783 } } peers { permanent_uuid: "70d9c6c1008544b4b7ba623a53d9e680" member_type: VOTER last_known_addr { host: "127.12.34.252" port: 44557 } } }
I20260926 15:09:34.390857 19310 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70d9c6c1008544b4b7ba623a53d9e680 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 15:09:34.391499 19304 sys_catalog.cc:564] T 00000000000000000000000000000000 P 70d9c6c1008544b4b7ba623a53d9e680 [sys.catalog]: configured and running, proceeding with master startup.
W20260926 15:09:34.400486 19321 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 70d9c6c1008544b4b7ba623a53d9e680: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 15:09:34.400624 19321 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 15:09:34.405613 19150 master_runner.cc:428] Detected that this master 70d9c6c1008544b4b7ba623a53d9e680 is joining an existing cluster
I20260926 15:09:34.405748 19150 master_runner.cc:432] Initiating AddMaster RPC to add 127.12.34.252:44557
I20260926 15:09:34.429905 19328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:34.409894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:42537 (local address 127.0.0.1:36794)
0926 15:09:34.413538 (+ 3644us) negotiation.cc:107] Waiting for socket to connect
0926 15:09:34.413560 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 15:09:34.413755 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:09:34.414474 (+ 719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:09:34.414479 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:09:34.415188 (+ 709us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:09:34.416099 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:34.416112 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:34.417380 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:34.417385 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:34.419280 (+ 1895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:34.419293 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:34.420435 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:34.420443 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:34.420615 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:34.421382 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:09:34.421414 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:09:34.422682 (+ 1268us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:09:34.426471 (+ 3789us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:09:34.426477 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:09:34.426481 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:09:34.426923 (+ 442us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:09:34.427415 (+ 492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:09:34.427420 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:09:34.427427 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:09:34.427642 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:09:34.428367 (+ 725us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:09:34.428372 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:09:34.428668 (+ 296us) client_negotiation.cc:769] Sending connection context
0926 15:09:34.428814 (+ 146us) client_negotiation.cc:240] Negotiation successful
0926 15:09:34.429406 (+ 592us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3507,"thread_start_us":3868,"threads_started":1}
I20260926 15:09:34.430109 19240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:34.409964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44524 (local address 127.12.34.253:32783)
0926 15:09:34.410293 (+ 329us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:34.410299 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:34.414188 (+ 3889us) server_negotiation.cc:407] Connection header received
0926 15:09:34.414429 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:34.414435 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:34.414529 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:34.414620 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:09:34.416921 (+ 2301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:34.417988 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:34.419874 (+ 1886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:34.420540 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:34.420629 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:34.423321 (+ 2692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:09:34.423329 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:09:34.423336 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:09:34.423374 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:09:34.427121 (+ 3747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:34.427905 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:34.427910 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:34.427914 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:34.428049 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:34.428499 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:34.428504 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:34.428507 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:34.428848 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:09:34.428956 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 15:09:34.429542 (+ 586us) server_negotiation.cc:299] Negotiation successful
0926 15:09:34.429773 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":227}
I20260926 15:09:34.430841 19120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:34.409730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36794 (local address 127.12.34.254:42537)
0926 15:09:34.410163 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:34.410170 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:34.413640 (+ 3470us) server_negotiation.cc:407] Connection header received
0926 15:09:34.413919 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:34.413925 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:34.414012 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:34.414088 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:09:34.416250 (+ 2162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:34.417220 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:34.419444 (+ 2224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:34.420103 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:34.420211 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:34.422850 (+ 2639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:09:34.422857 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:09:34.422861 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:09:34.422904 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:09:34.426300 (+ 3396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:34.427073 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:34.427080 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:34.427084 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:34.427256 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:34.427815 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:34.427820 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:34.427824 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:34.428144 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:09:34.428292 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0926 15:09:34.429021 (+ 729us) server_negotiation.cc:299] Negotiation successful
0926 15:09:34.430510 (+ 1489us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296}
I20260926 15:09:34.431177 19327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:34.409921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.253:32783 (local address 127.0.0.1:44524)
0926 15:09:34.414095 (+ 4174us) negotiation.cc:107] Waiting for socket to connect
0926 15:09:34.414111 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 15:09:34.414280 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:09:34.414910 (+ 630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:09:34.414917 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:09:34.416339 (+ 1422us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:09:34.416748 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:34.416764 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:34.418166 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:34.418172 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:34.419726 (+ 1554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:34.419737 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:34.420898 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:34.420906 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:34.421119 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:34.421734 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:09:34.421759 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:09:34.423145 (+ 1386us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:09:34.427282 (+ 4137us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:09:34.427287 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:09:34.427290 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:09:34.427760 (+ 470us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:09:34.428189 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:09:34.428193 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:09:34.428196 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:09:34.428367 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:09:34.429037 (+ 670us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:09:34.429041 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:09:34.429249 (+ 208us) client_negotiation.cc:769] Sending connection context
0926 15:09:34.430354 (+ 1105us) client_negotiation.cc:240] Negotiation successful
0926 15:09:34.430695 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4035,"thread_start_us":3338,"threads_started":1}
I20260926 15:09:34.438661 19329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:34.414393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.252:44557 (local address 127.0.0.1:54098)
0926 15:09:34.422842 (+ 8449us) negotiation.cc:107] Waiting for socket to connect
0926 15:09:34.422862 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 15:09:34.422991 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:09:34.423410 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:09:34.423417 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:09:34.425231 (+ 1814us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:09:34.425589 (+ 358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:34.425602 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:34.427087 (+ 1485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:34.427092 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:34.428670 (+ 1578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:34.428682 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:34.429867 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:34.429874 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:34.430034 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:34.430621 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:09:34.430648 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:09:34.431788 (+ 1140us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:09:34.435945 (+ 4157us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:09:34.435952 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:09:34.435957 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:09:34.436302 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:09:34.436647 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:09:34.436655 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:09:34.436658 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:09:34.436804 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:09:34.437563 (+ 759us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:09:34.437567 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:09:34.437734 (+ 167us) client_negotiation.cc:769] Sending connection context
0926 15:09:34.437891 (+ 157us) client_negotiation.cc:240] Negotiation successful
0926 15:09:34.438173 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8326,"thread_start_us":8194,"threads_started":1}
I20260926 15:09:34.438728 19307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:34.414219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54098 (local address 127.12.34.252:44557)
0926 15:09:34.414406 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:34.414412 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:34.422920 (+ 8508us) server_negotiation.cc:407] Connection header received
0926 15:09:34.423105 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:34.423110 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:34.423166 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:34.423238 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:09:34.425740 (+ 2502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:34.426855 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:34.428800 (+ 1945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:34.429503 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:34.429593 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:34.431966 (+ 2373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:09:34.431971 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:09:34.431975 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:09:34.432006 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:09:34.435825 (+ 3819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:34.436414 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:34.436421 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:34.436424 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:34.436545 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:34.436943 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:34.436951 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:34.436956 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:34.437300 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:09:34.437467 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0926 15:09:34.438015 (+ 548us) server_negotiation.cc:299] Negotiation successful
0926 15:09:34.438332 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":106,"server-negotiator.queue_time_us":38}
Sep 26 15:09:34 dist-test-slave-865v krb5kdc[18543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435371, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.34.254@KRBTEST.COM for oryx/127.12.34.252@KRBTEST.COM
I20260926 15:09:34.445994 12427 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 18649
I20260926 15:09:34.451699 19143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:34.436628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.252:44557 (local address 127.0.0.1:54102)
0926 15:09:34.436795 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0926 15:09:34.436813 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 15:09:34.436911 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:09:34.438096 (+ 1185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:09:34.438101 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:09:34.438525 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:09:34.438716 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:34.438726 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:34.440507 (+ 1781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:34.440513 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:34.441420 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:34.441431 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:34.442150 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:34.442156 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:34.442313 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:34.442824 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:09:34.442854 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:09:34.445228 (+ 2374us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:09:34.449128 (+ 3900us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:09:34.449134 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:09:34.449138 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:09:34.449606 (+ 468us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:09:34.450024 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:09:34.450029 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:09:34.450033 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:09:34.450179 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:09:34.450774 (+ 595us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:09:34.450779 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:09:34.450928 (+ 149us) client_negotiation.cc:769] Sending connection context
0926 15:09:34.451069 (+ 141us) client_negotiation.cc:240] Negotiation successful
0926 15:09:34.451325 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260926 15:09:34.452557 19330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:34.437052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54102 (local address 127.12.34.252:44557)
0926 15:09:34.437750 (+ 698us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:34.437758 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:34.437777 (+ 19us) server_negotiation.cc:407] Connection header received
0926 15:09:34.437851 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:34.437859 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:34.437934 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:34.438855 (+ 921us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:09:34.439096 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:34.440324 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:34.441559 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:34.442012 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:34.442143 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:34.445449 (+ 3306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:09:34.445455 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:09:34.445460 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:09:34.445495 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:09:34.448949 (+ 3454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:34.449731 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:34.449735 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:34.449740 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:34.449903 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:34.450309 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:34.450314 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:34.450318 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:34.450643 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:09:34.450778 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0926 15:09:34.451979 (+ 1201us) server_negotiation.cc:299] Negotiation successful
0926 15:09:34.452173 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":566,"thread_start_us":382,"threads_started":1}
I20260926 15:09:34.453904 18587 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.12.34.252:44557: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "70d9c6c1008544b4b7ba623a53d9e680"
member_type: NON_VOTER
last_known_addr {
host: "127.12.34.252"
port: 44557
}
attrs {
promote: true
}
}
dest_uuid: "f8c077ce1f004166989faaf5668ef1e8"
cas_config_opid_index: 5
I20260926 15:09:34.454736 18587 consensus_queue.cc:237] T 00000000000000000000000000000000 P f8c077ce1f004166989faaf5668ef1e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f8c077ce1f004166989faaf5668ef1e8" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 42537 } } peers { permanent_uuid: "f183e2953d3141608ce600f9167cd229" member_type: NON_VOTER last_known_addr { host: "127.12.34.253" port: 32783 } attrs { promote: true } } peers { permanent_uuid: "70d9c6c1008544b4b7ba623a53d9e680" member_type: NON_VOTER last_known_addr { host: "127.12.34.252" port: 44557 } attrs { promote: true } }
I20260926 15:09:34.456377 19146 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f8c077ce1f004166989faaf5668ef1e8 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 70d9c6c1008544b4b7ba623a53d9e680 (127.12.34.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f8c077ce1f004166989faaf5668ef1e8" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 42537 } } peers { permanent_uuid: "f183e2953d3141608ce600f9167cd229" member_type: NON_VOTER last_known_addr { host: "127.12.34.253" port: 32783 } attrs { promote: true } } peers { permanent_uuid: "70d9c6c1008544b4b7ba623a53d9e680" member_type: NON_VOTER last_known_addr { host: "127.12.34.252" port: 44557 } attrs { promote: true } } }
Sep 26 15:09:34 dist-test-slave-865v krb5kdc[18543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435371, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.12.34.193@KRBTEST.COM
I20260926 15:09:34.461843 19146 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.12.34.252:44557
I20260926 15:09:34.461907 19197 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f183e2953d3141608ce600f9167cd229 [term 1 LEARNER]: Refusing update from remote peer f8c077ce1f004166989faaf5668ef1e8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260926 15:09:34.462604 19150 master.cc:537] Master@127.12.34.252:44557 shutting down...
I20260926 15:09:34.467249 19331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:34.446880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57008 (local address 127.12.34.193:38979)
0926 15:09:34.447516 (+ 636us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:34.447525 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:34.447554 (+ 29us) server_negotiation.cc:407] Connection header received
0926 15:09:34.447617 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:34.447625 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:34.447888 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:34.448047 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:09:34.449562 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:34.450971 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:34.452344 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:34.453065 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:34.453197 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:34.459014 (+ 5817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:09:34.459026 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:09:34.459044 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:09:34.459084 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:09:34.462761 (+ 3677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:34.463571 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:34.463579 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:34.463583 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:34.463721 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:34.464146 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:34.464151 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:34.464154 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:34.464544 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:09:34.466183 (+ 1639us) server_negotiation.cc:1035] Waiting for connection context
0926 15:09:34.466363 (+ 180us) server_negotiation.cc:299] Negotiation successful
0926 15:09:34.466685 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":462,"thread_start_us":336,"threads_started":1}
I20260926 15:09:34.468806 18772 generic_service.cc:196] Checking for leaks (request via RPC)
W20260926 15:09:34.475152 18574 consensus_peers.cc:487] T 00000000000000000000000000000000 P f8c077ce1f004166989faaf5668ef1e8 -> Peer 70d9c6c1008544b4b7ba623a53d9e680 (127.12.34.252:44557): Couldn't send request to peer 70d9c6c1008544b4b7ba623a53d9e680. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260926 15:09:34.481940 19336 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f8c077ce1f004166989faaf5668ef1e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f183e2953d3141608ce600f9167cd229" member_type: NON_VOTER last_known_addr { host: "127.12.34.253" port: 32783 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260926 15:09:34.491688 19332 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8c077ce1f004166989faaf5668ef1e8 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "f8c077ce1f004166989faaf5668ef1e8" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f8c077ce1f004166989faaf5668ef1e8" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 42537 } } peers { permanent_uuid: "f183e2953d3141608ce600f9167cd229" member_type: NON_VOTER last_known_addr { host: "127.12.34.253" port: 32783 } attrs { promote: true } } peers { permanent_uuid: "70d9c6c1008544b4b7ba623a53d9e680" member_type: NON_VOTER last_known_addr { host: "127.12.34.252" port: 44557 } attrs { promote: true } } }
I20260926 15:09:34.492182 19332 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8c077ce1f004166989faaf5668ef1e8 [sys.catalog]: This master's current role is: LEADER
I20260926 15:09:34.495404 19197 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f183e2953d3141608ce600f9167cd229 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 70d9c6c1008544b4b7ba623a53d9e680 (127.12.34.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f8c077ce1f004166989faaf5668ef1e8" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 42537 } } peers { permanent_uuid: "f183e2953d3141608ce600f9167cd229" member_type: NON_VOTER last_known_addr { host: "127.12.34.253" port: 32783 } attrs { promote: true } } peers { permanent_uuid: "70d9c6c1008544b4b7ba623a53d9e680" member_type: NON_VOTER last_known_addr { host: "127.12.34.252" port: 44557 } attrs { promote: true } } }
I20260926 15:09:34.500892 19225 sys_catalog.cc:455] T 00000000000000000000000000000000 P f183e2953d3141608ce600f9167cd229 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f8c077ce1f004166989faaf5668ef1e8" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f8c077ce1f004166989faaf5668ef1e8" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 42537 } } peers { permanent_uuid: "f183e2953d3141608ce600f9167cd229" member_type: NON_VOTER last_known_addr { host: "127.12.34.253" port: 32783 } attrs { promote: true } } peers { permanent_uuid: "70d9c6c1008544b4b7ba623a53d9e680" member_type: NON_VOTER last_known_addr { host: "127.12.34.252" port: 44557 } attrs { promote: true } } }
I20260926 15:09:34.501205 19225 sys_catalog.cc:458] T 00000000000000000000000000000000 P f183e2953d3141608ce600f9167cd229 [sys.catalog]: This master's current role is: LEARNER
I20260926 15:09:34.501569 19333 raft_consensus.cc:1050] T 00000000000000000000000000000000 P f8c077ce1f004166989faaf5668ef1e8: attempting to promote NON_VOTER f183e2953d3141608ce600f9167cd229 to VOTER
I20260926 15:09:34.502490 19333 consensus_queue.cc:237] T 00000000000000000000000000000000 P f8c077ce1f004166989faaf5668ef1e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f8c077ce1f004166989faaf5668ef1e8" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 42537 } } peers { permanent_uuid: "f183e2953d3141608ce600f9167cd229" member_type: VOTER last_known_addr { host: "127.12.34.253" port: 32783 } attrs { promote: false } } peers { permanent_uuid: "70d9c6c1008544b4b7ba623a53d9e680" member_type: NON_VOTER last_known_addr { host: "127.12.34.252" port: 44557 } attrs { promote: true } }
I20260926 15:09:34.504937 19197 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f183e2953d3141608ce600f9167cd229 [term 1 LEARNER]: Refusing update from remote peer f8c077ce1f004166989faaf5668ef1e8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260926 15:09:34.505453 19336 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f8c077ce1f004166989faaf5668ef1e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f183e2953d3141608ce600f9167cd229" member_type: VOTER last_known_addr { host: "127.12.34.253" port: 32783 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260926 15:09:34.507807 18574 consensus_peers.cc:487] T 00000000000000000000000000000000 P f8c077ce1f004166989faaf5668ef1e8 -> Peer 70d9c6c1008544b4b7ba623a53d9e680 (127.12.34.252:44557): Couldn't send request to peer 70d9c6c1008544b4b7ba623a53d9e680. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260926 15:09:34.508885 19332 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f8c077ce1f004166989faaf5668ef1e8 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, f183e2953d3141608ce600f9167cd229 (127.12.34.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f8c077ce1f004166989faaf5668ef1e8" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 42537 } } peers { permanent_uuid: "f183e2953d3141608ce600f9167cd229" member_type: VOTER last_known_addr { host: "127.12.34.253" port: 32783 } attrs { promote: false } } peers { permanent_uuid: "70d9c6c1008544b4b7ba623a53d9e680" member_type: NON_VOTER last_known_addr { host: "127.12.34.252" port: 44557 } attrs { promote: true } } }
I20260926 15:09:34.510072 19150 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 70d9c6c1008544b4b7ba623a53d9e680 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260926 15:09:34.510478 19150 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 70d9c6c1008544b4b7ba623a53d9e680 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260926 15:09:34.510674 19150 tablet_replica.cc:330] T 00000000000000000000000000000000 P 70d9c6c1008544b4b7ba623a53d9e680: stopping tablet replica
I20260926 15:09:34.513286 19146 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8c077ce1f004166989faaf5668ef1e8 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "f8c077ce1f004166989faaf5668ef1e8" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f8c077ce1f004166989faaf5668ef1e8" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 42537 } } peers { permanent_uuid: "f183e2953d3141608ce600f9167cd229" member_type: VOTER last_known_addr { host: "127.12.34.253" port: 32783 } attrs { promote: false } } peers { permanent_uuid: "70d9c6c1008544b4b7ba623a53d9e680" member_type: NON_VOTER last_known_addr { host: "127.12.34.252" port: 44557 } attrs { promote: true } } }
I20260926 15:09:34.513532 19146 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8c077ce1f004166989faaf5668ef1e8 [sys.catalog]: This master's current role is: LEADER
I20260926 15:09:34.513927 19146 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8c077ce1f004166989faaf5668ef1e8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f8c077ce1f004166989faaf5668ef1e8" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f8c077ce1f004166989faaf5668ef1e8" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 42537 } } peers { permanent_uuid: "f183e2953d3141608ce600f9167cd229" member_type: VOTER last_known_addr { host: "127.12.34.253" port: 32783 } attrs { promote: false } } peers { permanent_uuid: "70d9c6c1008544b4b7ba623a53d9e680" member_type: NON_VOTER last_known_addr { host: "127.12.34.252" port: 44557 } attrs { promote: true } } }
I20260926 15:09:34.513662 19196 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f183e2953d3141608ce600f9167cd229 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, f183e2953d3141608ce600f9167cd229 (127.12.34.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f8c077ce1f004166989faaf5668ef1e8" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 42537 } } peers { permanent_uuid: "f183e2953d3141608ce600f9167cd229" member_type: VOTER last_known_addr { host: "127.12.34.253" port: 32783 } attrs { promote: false } } peers { permanent_uuid: "70d9c6c1008544b4b7ba623a53d9e680" member_type: NON_VOTER last_known_addr { host: "127.12.34.252" port: 44557 } attrs { promote: true } } }
I20260926 15:09:34.514163 19146 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8c077ce1f004166989faaf5668ef1e8 [sys.catalog]: This master's current role is: LEADER
I20260926 15:09:34.516685 19225 sys_catalog.cc:455] T 00000000000000000000000000000000 P f183e2953d3141608ce600f9167cd229 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f8c077ce1f004166989faaf5668ef1e8" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f8c077ce1f004166989faaf5668ef1e8" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 42537 } } peers { permanent_uuid: "f183e2953d3141608ce600f9167cd229" member_type: VOTER last_known_addr { host: "127.12.34.253" port: 32783 } attrs { promote: false } } peers { permanent_uuid: "70d9c6c1008544b4b7ba623a53d9e680" member_type: NON_VOTER last_known_addr { host: "127.12.34.252" port: 44557 } attrs { promote: true } } }
I20260926 15:09:34.516935 19225 sys_catalog.cc:458] T 00000000000000000000000000000000 P f183e2953d3141608ce600f9167cd229 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 15:09:34.531448 19150 master.cc:552] Master@127.12.34.252:44557 shutdown complete.
I20260926 15:09:34.531559 19150 master_runner.cc:305] Clearing existing system tablet
I20260926 15:09:34.531893 19150 fs_manager.cc:359] Metadata directory not provided
I20260926 15:09:34.532006 19150 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:09:34.534284 19150 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260926 15:09:34.535712 19340 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:09:34.536027 19150 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:09:34.536286 19150 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 15:09:34.536381 19150 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/minicluster-data/master-2/data,/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/minicluster-data/master-2/wal
uuid: "70d9c6c1008544b4b7ba623a53d9e680"
format_stamp: "Formatted at 2026-09-26 15:09:34 on dist-test-slave-865v"
server_key: "2ef40050b8fc7f4a02a491d85060f093"
server_key_iv: "89d3fc1a788098cfba0e00eb187c0532"
server_key_version: "encryptionkey@0"
I20260926 15:09:34.539758 19150 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 70d9c6c1008544b4b7ba623a53d9e680: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:09:34.544515 19150 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 70d9c6c1008544b4b7ba623a53d9e680: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260926 15:09:34.544680 19150 log.cc:1196] T 00000000000000000000000000000000 P 70d9c6c1008544b4b7ba623a53d9e680: Deleting WAL directory at /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260926 15:09:34.545069 19150 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 70d9c6c1008544b4b7ba623a53d9e680: Deleting consensus metadata
I20260926 15:09:34.545440 19150 master_runner.cc:315] Copying system tablet from 127.12.34.254:42537
I20260926 15:09:34.548024 19150 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 70d9c6c1008544b4b7ba623a53d9e680: tablet copy: Beginning tablet copy session from remote peer at address 127.12.34.254:42537
I20260926 15:09:34.559957 19120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:34.548773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36810 (local address 127.12.34.254:42537)
0926 15:09:34.549010 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:34.549020 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:34.549517 (+ 497us) server_negotiation.cc:407] Connection header received
0926 15:09:34.549722 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:34.549727 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:34.549801 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:34.549912 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:09:34.550665 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:34.551365 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:34.552547 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:34.552944 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:34.553081 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:34.554871 (+ 1790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:09:34.554877 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:09:34.554882 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:09:34.554923 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:09:34.557466 (+ 2543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:34.558069 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:34.558079 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:34.558082 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:34.558195 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:34.558565 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:34.558570 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:34.558574 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:34.558824 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:09:34.558974 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0926 15:09:34.559383 (+ 409us) server_negotiation.cc:299] Negotiation successful
0926 15:09:34.559596 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260926 15:09:34.560009 19346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:34.548772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:42537 (local address 127.0.0.1:36810)
0926 15:09:34.549412 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0926 15:09:34.549428 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 15:09:34.549585 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:09:34.549923 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:09:34.549927 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:09:34.550342 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:09:34.550523 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:34.550531 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:34.551491 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:34.551499 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:34.552407 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:34.552418 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:34.553124 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:34.553130 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:34.553323 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:34.553968 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:09:34.553985 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:09:34.554674 (+ 689us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:09:34.557590 (+ 2916us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:09:34.557598 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:09:34.557602 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:09:34.557950 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:09:34.558300 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:09:34.558307 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:09:34.558310 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:09:34.558454 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:09:34.558951 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:09:34.558956 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:09:34.559111 (+ 155us) client_negotiation.cc:769] Sending connection context
0926 15:09:34.559260 (+ 149us) client_negotiation.cc:240] Negotiation successful
0926 15:09:34.559588 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":366,"threads_started":1}
I20260926 15:09:34.560788 18608 tablet_copy_service.cc:140] P f8c077ce1f004166989faaf5668ef1e8: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 70d9c6c1008544b4b7ba623a53d9e680 ({username='oryx', principal='oryx/127.12.34.252@KRBTEST.COM'} at 127.0.0.1:36810)
I20260926 15:09:34.560961 18608 tablet_copy_service.cc:161] P f8c077ce1f004166989faaf5668ef1e8: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 70d9c6c1008544b4b7ba623a53d9e680 at {username='oryx', principal='oryx/127.12.34.252@KRBTEST.COM'} at 127.0.0.1:36810: session id = 70d9c6c1008544b4b7ba623a53d9e680-00000000000000000000000000000000
I20260926 15:09:34.563138 18608 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P f8c077ce1f004166989faaf5668ef1e8: Tablet Copy: opened 0 blocks and 1 log segments
I20260926 15:09:34.564608 19150 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260926 15:09:34.572557 19150 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 70d9c6c1008544b4b7ba623a53d9e680: tablet copy: Starting download of 0 data blocks...
I20260926 15:09:34.572819 19150 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 70d9c6c1008544b4b7ba623a53d9e680: tablet copy: Starting download of 1 WAL segments...
I20260926 15:09:34.576943 19150 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 70d9c6c1008544b4b7ba623a53d9e680: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260926 15:09:34.580010 18608 tablet_copy_service.cc:342] P f8c077ce1f004166989faaf5668ef1e8: Request end of tablet copy session 70d9c6c1008544b4b7ba623a53d9e680-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.12.34.252@KRBTEST.COM'} at 127.0.0.1:36810
I20260926 15:09:34.580135 18608 tablet_copy_service.cc:433] P f8c077ce1f004166989faaf5668ef1e8: ending tablet copy session 70d9c6c1008544b4b7ba623a53d9e680-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 70d9c6c1008544b4b7ba623a53d9e680
I20260926 15:09:34.590291 19150 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:09:34.618822 19347 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:09:34.624935 19348 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:09:34.627677 12427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8uoJGv/build/asan/bin/kudu with pid 18649
W20260926 15:09:34.641366 19350 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:09:34.641486 19150 server_base.cc:1034] running on GCE node
I20260926 15:09:34.641985 19150 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:09:34.649336 19150 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:09:34.650544 19150 hybrid_clock.cc:648] HybridClock initialized: now 1790435374650466 us; error 80 us; skew 500 ppm
I20260926 15:09:34.650822 19150 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 15:09:34.651997 12427 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8uoJGv/build/asan/bin/kudu
/tmp/dist-test-task8uoJGv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.34.193:38979
--local_ip_for_outbound_sockets=127.12.34.193
--tserver_master_addrs=127.12.34.254:42537
--webserver_port=40817
--webserver_interface=127.12.34.193
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/krb5kdc/oryx_127.12.34.193.keytab
--principal=oryx/127.12.34.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.34.212:35299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.12.34.254:42537,127.12.34.253:32783
--tserver_master_addrs=127.12.34.254:42537,127.12.34.253:32783,127.12.34.252:44557 with env {KRB5CCNAME=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 15:09:34 dist-test-slave-865v krb5kdc[18543](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435374, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.34.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:09:34.652805 19150 init.cc:377] Logged in from keytab as oryx/127.12.34.252@KRBTEST.COM (short username oryx)
I20260926 15:09:34.657160 19150 webserver.cc:460] Webserver started at http://127.12.34.252:45865/ using document root <none> and password file <none>
I20260926 15:09:34.657469 19150 fs_manager.cc:359] Metadata directory not provided
I20260926 15:09:34.657577 19150 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:09:34.660446 19150 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260926 15:09:34.662081 19358 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:09:34.664629 19150 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 15:09:34.664755 19150 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/minicluster-data/master-2/data,/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/minicluster-data/master-2/wal
uuid: "70d9c6c1008544b4b7ba623a53d9e680"
format_stamp: "Formatted at 2026-09-26 15:09:34 on dist-test-slave-865v"
server_key: "2ef40050b8fc7f4a02a491d85060f093"
server_key_iv: "89d3fc1a788098cfba0e00eb187c0532"
server_key_version: "encryptionkey@0"
I20260926 15:09:34.664948 19150 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:09:34.685961 19150 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:09:34.689278 19150 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:09:34.734179 19150 rpc_server.cc:307] RPC server started. Bound to: 127.12.34.252:44557
I20260926 15:09:34.734218 19409 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.34.252:44557 every 8 connection(s)
I20260926 15:09:34.735185 19150 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/minicluster-data/master-2/data/info.pb
I20260926 15:09:34.742360 19410 sys_catalog.cc:263] Verifying existing consensus state
I20260926 15:09:34.748857 19410 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70d9c6c1008544b4b7ba623a53d9e680: Bootstrap starting.
I20260926 15:09:34.763597 19410 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70d9c6c1008544b4b7ba623a53d9e680: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 15:09:34.764114 19410 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70d9c6c1008544b4b7ba623a53d9e680: Bootstrap complete.
I20260926 15:09:34.765048 19410 raft_consensus.cc:348] T 00000000000000000000000000000000 P 70d9c6c1008544b4b7ba623a53d9e680 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f8c077ce1f004166989faaf5668ef1e8" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 42537 } } peers { permanent_uuid: "f183e2953d3141608ce600f9167cd229" member_type: VOTER last_known_addr { host: "127.12.34.253" port: 32783 } attrs { promote: false } } peers { permanent_uuid: "70d9c6c1008544b4b7ba623a53d9e680" member_type: NON_VOTER last_known_addr { host: "127.12.34.252" port: 44557 } attrs { promote: true } }
I20260926 15:09:34.765398 19410 raft_consensus.cc:729] T 00000000000000000000000000000000 P 70d9c6c1008544b4b7ba623a53d9e680 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 70d9c6c1008544b4b7ba623a53d9e680, State: Initialized, Role: LEARNER
I20260926 15:09:34.765746 19410 consensus_queue.cc:260] T 00000000000000000000000000000000 P 70d9c6c1008544b4b7ba623a53d9e680 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f8c077ce1f004166989faaf5668ef1e8" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 42537 } } peers { permanent_uuid: "f183e2953d3141608ce600f9167cd229" member_type: VOTER last_known_addr { host: "127.12.34.253" port: 32783 } attrs { promote: false } } peers { permanent_uuid: "70d9c6c1008544b4b7ba623a53d9e680" member_type: NON_VOTER last_known_addr { host: "127.12.34.252" port: 44557 } attrs { promote: true } }
I20260926 15:09:34.766556 19413 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70d9c6c1008544b4b7ba623a53d9e680 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f8c077ce1f004166989faaf5668ef1e8" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 42537 } } peers { permanent_uuid: "f183e2953d3141608ce600f9167cd229" member_type: VOTER last_known_addr { host: "127.12.34.253" port: 32783 } attrs { promote: false } } peers { permanent_uuid: "70d9c6c1008544b4b7ba623a53d9e680" member_type: NON_VOTER last_known_addr { host: "127.12.34.252" port: 44557 } attrs { promote: true } } }
I20260926 15:09:34.766958 19410 sys_catalog.cc:564] T 00000000000000000000000000000000 P 70d9c6c1008544b4b7ba623a53d9e680 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:09:34.766965 19413 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70d9c6c1008544b4b7ba623a53d9e680 [sys.catalog]: This master's current role is: LEARNER
I20260926 15:09:34.774518 19424 catalog_manager.cc:1259] Loaded cluster ID: 165375e664a14f989e39db1da9eac0b9
I20260926 15:09:34.774696 19424 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 70d9c6c1008544b4b7ba623a53d9e680: loading cluster ID for follower catalog manager: success
I20260926 15:09:34.777261 19424 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 70d9c6c1008544b4b7ba623a53d9e680: acquiring CA information for follower catalog manager: success
I20260926 15:09:34.779109 19424 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 70d9c6c1008544b4b7ba623a53d9e680: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260926 15:09:35.024402 19356 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:09:35.024662 19356 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:09:35.024763 19356 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:09:35.032248 19356 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:09:35.032346 19356 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:09:35.032452 19356 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.34.193
I20260926 15:09:35.042541 19356 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.34.212:35299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/krb5kdc/oryx_127.12.34.193.keytab
--principal=oryx/127.12.34.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.34.193:38979
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.34.193
--webserver_port=40817
--tserver_master_addrs=127.12.34.254:42537,127.12.34.253:32783,127.12.34.252:44557
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.34.193
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:09:35.043749 19356 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:09:35.044981 19356 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:09:35.058184 19429 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:09:35.058136 19430 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:09:35.058291 19432 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:09:35.059276 19356 server_base.cc:1034] running on GCE node
I20260926 15:09:35.060148 19356 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:09:35.061198 19356 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:09:35.062423 19356 hybrid_clock.cc:648] HybridClock initialized: now 1790435375062380 us; error 28 us; skew 500 ppm
I20260926 15:09:35.062764 19356 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:09:35 dist-test-slave-865v krb5kdc[18543](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435375, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.34.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:09:35.066365 19356 init.cc:377] Logged in from keytab as oryx/127.12.34.193@KRBTEST.COM (short username oryx)
I20260926 15:09:35.067919 19356 webserver.cc:460] Webserver started at http://127.12.34.193:40817/ using document root <none> and password file <none>
I20260926 15:09:35.068634 19356 fs_manager.cc:359] Metadata directory not provided
I20260926 15:09:35.068771 19356 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:09:35.074018 19356 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 15:09:35.077515 19439 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:09:35.079236 19356 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260926 15:09:35.079471 19356 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/minicluster-data/ts-0/data,/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/minicluster-data/ts-0/wal
uuid: "d3e156f17705476c90ae1f60578daa75"
format_stamp: "Formatted at 2026-09-26 15:09:32 on dist-test-slave-865v"
server_key: "3b8749c227651f4b35758c4e969c6d5a"
server_key_iv: "10a9d62ce1a7f9ea3c3c316d6c550d8c"
server_key_version: "encryptionkey@0"
I20260926 15:09:35.080456 19356 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:09:35.098686 19356 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:09:35.103060 19356 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:09:35.103341 19356 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:09:35.104354 19356 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:09:35.105754 19356 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:09:35.105886 19356 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:09:35.106009 19356 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:09:35.106096 19356 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:09:35.118603 19453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:35.111129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54106 (local address 127.12.34.252:44557)
0926 15:09:35.112008 (+ 879us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:35.112017 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:35.112044 (+ 27us) server_negotiation.cc:407] Connection header received
0926 15:09:35.112140 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:35.112148 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:35.112255 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:35.112411 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:09:35.112798 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:35.114143 (+ 1345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:35.116375 (+ 2232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:35.117601 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:35.117781 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:35.117986 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0926 15:09:35.118152 (+ 166us) server_negotiation.cc:299] Negotiation successful
0926 15:09:35.118279 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":715,"thread_start_us":586,"threads_started":1}
I20260926 15:09:35.118588 19451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:35.110748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.252:44557 (local address 127.0.0.1:54106)
0926 15:09:35.111188 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0926 15:09:35.111223 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 15:09:35.111378 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:09:35.112456 (+ 1078us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:09:35.112462 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:09:35.112494 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:09:35.112647 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:35.112655 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:35.114333 (+ 1678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:35.114341 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:35.116176 (+ 1835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:35.116196 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:35.117807 (+ 1611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:35.117813 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:35.117970 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:35.118006 (+ 36us) client_negotiation.cc:769] Sending connection context
0926 15:09:35.118131 (+ 125us) client_negotiation.cc:240] Negotiation successful
0926 15:09:35.118279 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":215,"threads_started":1}
I20260926 15:09:35.119886 19385 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 70d9c6c1008544b4b7ba623a53d9e680 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260926 15:09:35.120594 19413 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70d9c6c1008544b4b7ba623a53d9e680 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f8c077ce1f004166989faaf5668ef1e8. Latest consensus state: current_term: 1 leader_uuid: "f8c077ce1f004166989faaf5668ef1e8" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f8c077ce1f004166989faaf5668ef1e8" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 42537 } } peers { permanent_uuid: "f183e2953d3141608ce600f9167cd229" member_type: VOTER last_known_addr { host: "127.12.34.253" port: 32783 } attrs { promote: false } } peers { permanent_uuid: "70d9c6c1008544b4b7ba623a53d9e680" member_type: NON_VOTER last_known_addr { host: "127.12.34.252" port: 44557 } attrs { promote: true } } }
I20260926 15:09:35.120900 19413 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70d9c6c1008544b4b7ba623a53d9e680 [sys.catalog]: This master's current role is: LEARNER
I20260926 15:09:35.148739 19356 rpc_server.cc:307] RPC server started. Bound to: 127.12.34.193:38979
I20260926 15:09:35.148795 19553 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.34.193:38979 every 8 connection(s)
I20260926 15:09:35.150133 19356 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/minicluster-data/ts-0/data/info.pb
I20260926 15:09:35.157430 12427 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8uoJGv/build/asan/bin/kudu as pid 19356
I20260926 15:09:35.157753 12427 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 18783
Sep 26 15:09:35 dist-test-slave-865v krb5kdc[18543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435375, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.34.193@KRBTEST.COM for oryx/127.12.34.252@KRBTEST.COM
Sep 26 15:09:35 dist-test-slave-865v krb5kdc[18543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435375, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.34.193@KRBTEST.COM for oryx/127.12.34.253@KRBTEST.COM
I20260926 15:09:35.180433 19558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:35.152859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.193:41709 (local address 127.12.34.253:32783)
0926 15:09:35.153326 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:35.153332 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:35.154577 (+ 1245us) server_negotiation.cc:407] Connection header received
0926 15:09:35.159336 (+ 4759us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:35.159343 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:35.159452 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:35.159577 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:09:35.161199 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:35.162526 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:35.166129 (+ 3603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:35.167044 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:35.167169 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:35.174319 (+ 7150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:09:35.174325 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:09:35.174330 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:09:35.174370 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:09:35.177145 (+ 2775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:35.177880 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:35.177885 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:35.177889 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:35.178012 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:35.178456 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:35.178461 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:35.178464 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:35.178728 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:09:35.178895 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0926 15:09:35.179786 (+ 891us) server_negotiation.cc:299] Negotiation successful
0926 15:09:35.180004 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":225,"threads_started":1}
I20260926 15:09:35.182214 19453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:35.152947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.193:45221 (local address 127.12.34.252:44557)
0926 15:09:35.153173 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:35.153179 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:35.154733 (+ 1554us) server_negotiation.cc:407] Connection header received
0926 15:09:35.158410 (+ 3677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:35.158416 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:35.158504 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:35.158586 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:09:35.161154 (+ 2568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:35.162540 (+ 1386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:35.166108 (+ 3568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:35.166925 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:35.167039 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:35.174377 (+ 7338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:09:35.174386 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:09:35.174394 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:09:35.174445 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:09:35.179017 (+ 4572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:35.179982 (+ 965us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:35.179987 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:35.179990 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:35.180146 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:35.180579 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:35.180584 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:35.180587 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:35.180914 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:09:35.181051 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0926 15:09:35.181654 (+ 603us) server_negotiation.cc:299] Negotiation successful
0926 15:09:35.181886 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 15:09:35.182895 19559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:35.153704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.253:32783 (local address 127.12.34.193:41709)
0926 15:09:35.154418 (+ 714us) negotiation.cc:107] Waiting for socket to connect
0926 15:09:35.154459 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 15:09:35.158139 (+ 3680us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:09:35.159810 (+ 1671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:09:35.159817 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:09:35.160563 (+ 746us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:09:35.160985 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:35.161000 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:35.162753 (+ 1753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:35.162761 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:35.165909 (+ 3148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:35.165948 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:35.167625 (+ 1677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:35.167634 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:35.167890 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:35.168872 (+ 982us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:09:35.168912 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:09:35.174052 (+ 5140us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:09:35.177298 (+ 3246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:09:35.177307 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:09:35.177323 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:09:35.177754 (+ 431us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:09:35.178131 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:09:35.178135 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:09:35.178139 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:09:35.178339 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:09:35.178876 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:09:35.178883 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:09:35.179301 (+ 418us) client_negotiation.cc:769] Sending connection context
0926 15:09:35.180712 (+ 1411us) client_negotiation.cc:240] Negotiation successful
0926 15:09:35.181449 (+ 737us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"mutex_wait_us":75,"spinlock_wait_cycles":1211392,"thread_start_us":414,"threads_started":1}
I20260926 15:09:35.184098 19560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:35.154011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.252:44557 (local address 127.12.34.193:45221)
0926 15:09:35.154624 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0926 15:09:35.154647 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 15:09:35.157896 (+ 3249us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:09:35.159078 (+ 1182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:09:35.159093 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:09:35.159956 (+ 863us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:09:35.160912 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:35.160943 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:35.162957 (+ 2014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:35.162965 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:35.165917 (+ 2952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:35.165949 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:35.167480 (+ 1531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:35.167489 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:35.167843 (+ 354us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:35.169053 (+ 1210us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:09:35.169105 (+ 52us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:09:35.174128 (+ 5023us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:09:35.179185 (+ 5057us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:09:35.179189 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:09:35.179193 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:09:35.179844 (+ 651us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:09:35.180275 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:09:35.180280 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:09:35.180283 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:09:35.180452 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:09:35.181166 (+ 714us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:09:35.181171 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:09:35.181346 (+ 175us) client_negotiation.cc:769] Sending connection context
0926 15:09:35.182514 (+ 1168us) client_negotiation.cc:240] Negotiation successful
0926 15:09:35.182787 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"mutex_wait_us":118,"spinlock_wait_cycles":896,"thread_start_us":1302,"threads_started":1}
I20260926 15:09:35.184798 19556 heartbeater.cc:347] Connected to a master server at 127.12.34.253:32783
I20260926 15:09:35.185134 19556 heartbeater.cc:464] Registering TS with master...
I20260926 15:09:35.185940 19556 heartbeater.cc:511] Master 127.12.34.253:32783 requested a full tablet report, sending...
I20260926 15:09:35.186877 19554 heartbeater.cc:347] Connected to a master server at 127.12.34.252:44557
I20260926 15:09:35.187018 19554 heartbeater.cc:464] Registering TS with master...
I20260926 15:09:35.187446 19554 heartbeater.cc:511] Master 127.12.34.252:44557 requested a full tablet report, sending...
I20260926 15:09:35.188234 19187 ts_manager.cc:194] Registered new tserver with Master: d3e156f17705476c90ae1f60578daa75 (127.12.34.193:38979)
I20260926 15:09:35.189496 19375 ts_manager.cc:194] Registered new tserver with Master: d3e156f17705476c90ae1f60578daa75 (127.12.34.193:38979)
Sep 26 15:09:35 dist-test-slave-865v krb5kdc[18543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435375, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.34.193@KRBTEST.COM for oryx/127.12.34.254@KRBTEST.COM
Sep 26 15:09:35 dist-test-slave-865v krb5kdc[18543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435371, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.12.34.194@KRBTEST.COM
I20260926 15:09:35.197836 19562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:35.155823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.193:39085 (local address 127.12.34.254:42537)
0926 15:09:35.174224 (+ 18401us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:35.174232 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:35.180119 (+ 5887us) server_negotiation.cc:407] Connection header received
0926 15:09:35.180186 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:35.180193 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:35.180276 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:35.180370 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:09:35.181818 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:35.182990 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:35.185520 (+ 2530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:35.186248 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:35.186390 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:35.192502 (+ 6112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:09:35.192507 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:09:35.192511 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:09:35.192540 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:09:35.195193 (+ 2653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:35.195990 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:35.195995 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:35.195997 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:35.196100 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:35.196467 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:35.196472 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:35.196475 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:35.196726 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:09:35.196872 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0926 15:09:35.197265 (+ 393us) server_negotiation.cc:299] Negotiation successful
0926 15:09:35.197469 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18233,"thread_start_us":18430,"threads_started":1}
I20260926 15:09:35.197852 19561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:35.155715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:42537 (local address 127.12.34.193:39085)
0926 15:09:35.179889 (+ 24174us) negotiation.cc:107] Waiting for socket to connect
0926 15:09:35.179913 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 15:09:35.180018 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:09:35.180745 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:09:35.180749 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:09:35.181330 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:09:35.181655 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:35.181668 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:35.183144 (+ 1476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:35.183150 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:35.185373 (+ 2223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:35.185386 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:35.186469 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:35.186480 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:35.186660 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:35.190353 (+ 3693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:09:35.190380 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:09:35.192309 (+ 1929us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:09:35.195520 (+ 3211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:09:35.195523 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:09:35.195526 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:09:35.195856 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:09:35.196209 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:09:35.196214 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:09:35.196217 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:09:35.196360 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:09:35.196843 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:09:35.196848 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:09:35.196987 (+ 139us) client_negotiation.cc:769] Sending connection context
0926 15:09:35.197158 (+ 171us) client_negotiation.cc:240] Negotiation successful
0926 15:09:35.197433 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24033,"thread_start_us":24823,"threads_started":1}
I20260926 15:09:35.198637 19555 heartbeater.cc:347] Connected to a master server at 127.12.34.254:42537
I20260926 15:09:35.198781 19555 heartbeater.cc:464] Registering TS with master...
I20260926 15:09:35.199174 19555 heartbeater.cc:511] Master 127.12.34.254:42537 requested a full tablet report, sending...
I20260926 15:09:35.199545 19564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:35.158628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40454 (local address 127.12.34.194:38821)
0926 15:09:35.163724 (+ 5096us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:35.163736 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:35.179160 (+ 15424us) server_negotiation.cc:407] Connection header received
0926 15:09:35.179338 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:35.179349 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:35.179633 (+ 284us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:35.179806 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:09:35.182471 (+ 2665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:35.184818 (+ 2347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:35.186615 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:35.187352 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:35.188170 (+ 818us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:35.193645 (+ 5475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:09:35.193656 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:09:35.193668 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:09:35.193700 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:09:35.196580 (+ 2880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:35.197302 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:35.197311 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:35.197315 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:35.197441 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:35.197861 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:35.197865 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:35.197869 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:35.198292 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:09:35.198541 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0926 15:09:35.198912 (+ 371us) server_negotiation.cc:299] Negotiation successful
0926 15:09:35.199160 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4920,"thread_start_us":4770,"threads_started":1}
I20260926 15:09:35.200326 18587 ts_manager.cc:194] Re-registered known tserver with Master: d3e156f17705476c90ae1f60578daa75 (127.12.34.193:38979)
I20260926 15:09:35.200850 18906 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:09:35.201284 18587 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.12.34.193@KRBTEST.COM'} at 127.12.34.193:39085
I20260926 15:09:35.287537 12427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8uoJGv/build/asan/bin/kudu with pid 18783
I20260926 15:09:35.304055 12427 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8uoJGv/build/asan/bin/kudu
/tmp/dist-test-task8uoJGv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.34.194:38821
--local_ip_for_outbound_sockets=127.12.34.194
--tserver_master_addrs=127.12.34.254:42537
--webserver_port=46461
--webserver_interface=127.12.34.194
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/krb5kdc/oryx_127.12.34.194.keytab
--principal=oryx/127.12.34.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.34.212:35299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.12.34.254:42537,127.12.34.253:32783
--tserver_master_addrs=127.12.34.254:42537,127.12.34.253:32783,127.12.34.252:44557 with env {KRB5CCNAME=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:09:35.623507 19566 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:09:35.623795 19566 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:09:35.623955 19566 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:09:35.631165 19566 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:09:35.631311 19566 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:09:35.631479 19566 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.34.194
I20260926 15:09:35.644563 19566 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.34.212:35299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/krb5kdc/oryx_127.12.34.194.keytab
--principal=oryx/127.12.34.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.34.194:38821
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.34.194
--webserver_port=46461
--tserver_master_addrs=127.12.34.254:42537,127.12.34.253:32783,127.12.34.252:44557
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.34.194
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:09:35.645984 19566 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:09:35.647380 19566 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:09:35.660465 19572 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:09:35.660624 19574 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:09:35.660634 19571 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:09:35.661422 19566 server_base.cc:1034] running on GCE node
I20260926 15:09:35.661921 19566 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:09:35.662703 19566 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:09:35.663930 19566 hybrid_clock.cc:648] HybridClock initialized: now 1790435375663888 us; error 29 us; skew 500 ppm
I20260926 15:09:35.664264 19566 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:09:35 dist-test-slave-865v krb5kdc[18543](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435375, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.34.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:09:35.667553 19566 init.cc:377] Logged in from keytab as oryx/127.12.34.194@KRBTEST.COM (short username oryx)
I20260926 15:09:35.669111 19566 webserver.cc:460] Webserver started at http://127.12.34.194:46461/ using document root <none> and password file <none>
I20260926 15:09:35.669745 19566 fs_manager.cc:359] Metadata directory not provided
I20260926 15:09:35.669881 19566 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:09:35.674885 19566 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 15:09:35.677480 19581 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:09:35.678764 19332 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8c077ce1f004166989faaf5668ef1e8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f8c077ce1f004166989faaf5668ef1e8" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f8c077ce1f004166989faaf5668ef1e8" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 42537 } } peers { permanent_uuid: "f183e2953d3141608ce600f9167cd229" member_type: VOTER last_known_addr { host: "127.12.34.253" port: 32783 } attrs { promote: false } } peers { permanent_uuid: "70d9c6c1008544b4b7ba623a53d9e680" member_type: NON_VOTER last_known_addr { host: "127.12.34.252" port: 44557 } attrs { promote: true } } }
I20260926 15:09:35.679054 19332 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8c077ce1f004166989faaf5668ef1e8 [sys.catalog]: This master's current role is: LEADER
I20260926 15:09:35.679183 19566 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 15:09:35.680461 19582 raft_consensus.cc:1050] T 00000000000000000000000000000000 P f8c077ce1f004166989faaf5668ef1e8: attempting to promote NON_VOTER 70d9c6c1008544b4b7ba623a53d9e680 to VOTER
I20260926 15:09:35.679394 19566 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/minicluster-data/ts-1/data,/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/minicluster-data/ts-1/wal
uuid: "85dd0480684b43909497b35c05e43908"
format_stamp: "Formatted at 2026-09-26 15:09:32 on dist-test-slave-865v"
server_key: "652b4e46885e8b018d96ded4a20a3aa5"
server_key_iv: "14038bfe6efb1c5f73193d3d114c5b2c"
server_key_version: "encryptionkey@0"
I20260926 15:09:35.680852 19566 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:09:35.681185 19582 consensus_queue.cc:237] T 00000000000000000000000000000000 P f8c077ce1f004166989faaf5668ef1e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f8c077ce1f004166989faaf5668ef1e8" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 42537 } } peers { permanent_uuid: "f183e2953d3141608ce600f9167cd229" member_type: VOTER last_known_addr { host: "127.12.34.253" port: 32783 } attrs { promote: false } } peers { permanent_uuid: "70d9c6c1008544b4b7ba623a53d9e680" member_type: VOTER last_known_addr { host: "127.12.34.252" port: 44557 } attrs { promote: false } }
I20260926 15:09:35.683435 19385 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 70d9c6c1008544b4b7ba623a53d9e680 [term 1 LEARNER]: Refusing update from remote peer f8c077ce1f004166989faaf5668ef1e8: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260926 15:09:35.683558 19196 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f183e2953d3141608ce600f9167cd229 [term 1 FOLLOWER]: Refusing update from remote peer f8c077ce1f004166989faaf5668ef1e8: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260926 15:09:35.684213 19582 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f8c077ce1f004166989faaf5668ef1e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "70d9c6c1008544b4b7ba623a53d9e680" member_type: VOTER last_known_addr { host: "127.12.34.252" port: 44557 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260926 15:09:35.684595 19583 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f8c077ce1f004166989faaf5668ef1e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f183e2953d3141608ce600f9167cd229" member_type: VOTER last_known_addr { host: "127.12.34.253" port: 32783 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260926 15:09:35.687870 19582 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f8c077ce1f004166989faaf5668ef1e8 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 70d9c6c1008544b4b7ba623a53d9e680 (127.12.34.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f8c077ce1f004166989faaf5668ef1e8" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 42537 } } peers { permanent_uuid: "f183e2953d3141608ce600f9167cd229" member_type: VOTER last_known_addr { host: "127.12.34.253" port: 32783 } attrs { promote: false } } peers { permanent_uuid: "70d9c6c1008544b4b7ba623a53d9e680" member_type: VOTER last_known_addr { host: "127.12.34.252" port: 44557 } attrs { promote: false } } }
I20260926 15:09:35.688562 19196 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f183e2953d3141608ce600f9167cd229 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 70d9c6c1008544b4b7ba623a53d9e680 (127.12.34.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f8c077ce1f004166989faaf5668ef1e8" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 42537 } } peers { permanent_uuid: "f183e2953d3141608ce600f9167cd229" member_type: VOTER last_known_addr { host: "127.12.34.253" port: 32783 } attrs { promote: false } } peers { permanent_uuid: "70d9c6c1008544b4b7ba623a53d9e680" member_type: VOTER last_known_addr { host: "127.12.34.252" port: 44557 } attrs { promote: false } } }
I20260926 15:09:35.693106 19582 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8c077ce1f004166989faaf5668ef1e8 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "f8c077ce1f004166989faaf5668ef1e8" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f8c077ce1f004166989faaf5668ef1e8" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 42537 } } peers { permanent_uuid: "f183e2953d3141608ce600f9167cd229" member_type: VOTER last_known_addr { host: "127.12.34.253" port: 32783 } attrs { promote: false } } peers { permanent_uuid: "70d9c6c1008544b4b7ba623a53d9e680" member_type: VOTER last_known_addr { host: "127.12.34.252" port: 44557 } attrs { promote: false } } }
I20260926 15:09:35.693446 19582 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8c077ce1f004166989faaf5668ef1e8 [sys.catalog]: This master's current role is: LEADER
I20260926 15:09:35.693751 19332 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8c077ce1f004166989faaf5668ef1e8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f8c077ce1f004166989faaf5668ef1e8" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f8c077ce1f004166989faaf5668ef1e8" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 42537 } } peers { permanent_uuid: "f183e2953d3141608ce600f9167cd229" member_type: VOTER last_known_addr { host: "127.12.34.253" port: 32783 } attrs { promote: false } } peers { permanent_uuid: "70d9c6c1008544b4b7ba623a53d9e680" member_type: VOTER last_known_addr { host: "127.12.34.252" port: 44557 } attrs { promote: false } } }
I20260926 15:09:35.694056 19332 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8c077ce1f004166989faaf5668ef1e8 [sys.catalog]: This master's current role is: LEADER
I20260926 15:09:35.697160 19589 sys_catalog.cc:455] T 00000000000000000000000000000000 P f183e2953d3141608ce600f9167cd229 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f8c077ce1f004166989faaf5668ef1e8" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f8c077ce1f004166989faaf5668ef1e8" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 42537 } } peers { permanent_uuid: "f183e2953d3141608ce600f9167cd229" member_type: VOTER last_known_addr { host: "127.12.34.253" port: 32783 } attrs { promote: false } } peers { permanent_uuid: "70d9c6c1008544b4b7ba623a53d9e680" member_type: VOTER last_known_addr { host: "127.12.34.252" port: 44557 } attrs { promote: false } } }
I20260926 15:09:35.697619 19589 sys_catalog.cc:458] T 00000000000000000000000000000000 P f183e2953d3141608ce600f9167cd229 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 15:09:35.699590 19583 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8c077ce1f004166989faaf5668ef1e8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f8c077ce1f004166989faaf5668ef1e8" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f8c077ce1f004166989faaf5668ef1e8" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 42537 } } peers { permanent_uuid: "f183e2953d3141608ce600f9167cd229" member_type: VOTER last_known_addr { host: "127.12.34.253" port: 32783 } attrs { promote: false } } peers { permanent_uuid: "70d9c6c1008544b4b7ba623a53d9e680" member_type: VOTER last_known_addr { host: "127.12.34.252" port: 44557 } attrs { promote: false } } }
I20260926 15:09:35.699800 19583 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8c077ce1f004166989faaf5668ef1e8 [sys.catalog]: This master's current role is: LEADER
I20260926 15:09:35.700524 19385 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 70d9c6c1008544b4b7ba623a53d9e680 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 70d9c6c1008544b4b7ba623a53d9e680 (127.12.34.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f8c077ce1f004166989faaf5668ef1e8" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 42537 } } peers { permanent_uuid: "f183e2953d3141608ce600f9167cd229" member_type: VOTER last_known_addr { host: "127.12.34.253" port: 32783 } attrs { promote: false } } peers { permanent_uuid: "70d9c6c1008544b4b7ba623a53d9e680" member_type: VOTER last_known_addr { host: "127.12.34.252" port: 44557 } attrs { promote: false } } }
I20260926 15:09:35.703678 19593 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70d9c6c1008544b4b7ba623a53d9e680 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f8c077ce1f004166989faaf5668ef1e8" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f8c077ce1f004166989faaf5668ef1e8" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 42537 } } peers { permanent_uuid: "f183e2953d3141608ce600f9167cd229" member_type: VOTER last_known_addr { host: "127.12.34.253" port: 32783 } attrs { promote: false } } peers { permanent_uuid: "70d9c6c1008544b4b7ba623a53d9e680" member_type: VOTER last_known_addr { host: "127.12.34.252" port: 44557 } attrs { promote: false } } }
I20260926 15:09:35.704005 19593 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70d9c6c1008544b4b7ba623a53d9e680 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 15:09:35.714723 19566 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:09:35.718897 19566 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:09:35.719189 19566 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:09:35.720252 19566 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:09:35.721747 19566 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:09:35.721832 19566 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:09:35.721966 19566 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:09:35.722044 19566 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:09:35.757826 19566 rpc_server.cc:307] RPC server started. Bound to: 127.12.34.194:38821
I20260926 15:09:35.757870 19701 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.34.194:38821 every 8 connection(s)
I20260926 15:09:35.759208 19566 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/minicluster-data/ts-1/data/info.pb
I20260926 15:09:35.763375 12427 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8uoJGv/build/asan/bin/kudu as pid 19566
I20260926 15:09:35.763674 12427 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 18916
Sep 26 15:09:35 dist-test-slave-865v krb5kdc[18543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435371, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.12.34.195@KRBTEST.COM
Sep 26 15:09:35 dist-test-slave-865v krb5kdc[18543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435375, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.34.194@KRBTEST.COM for oryx/127.12.34.252@KRBTEST.COM
Sep 26 15:09:35 dist-test-slave-865v krb5kdc[18543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435375, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.34.194@KRBTEST.COM for oryx/127.12.34.253@KRBTEST.COM
Sep 26 15:09:35 dist-test-slave-865v krb5kdc[18543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435375, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.34.194@KRBTEST.COM for oryx/127.12.34.254@KRBTEST.COM
I20260926 15:09:35.814081 19711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:35.764786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38922 (local address 127.12.34.195:36967)
0926 15:09:35.788298 (+ 23512us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:35.788312 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:35.791099 (+ 2787us) server_negotiation.cc:407] Connection header received
0926 15:09:35.791192 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:35.791204 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:35.791527 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:35.791700 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:09:35.794291 (+ 2591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:35.795648 (+ 1357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:35.797455 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:35.798247 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:35.798377 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:35.804559 (+ 6182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:09:35.804574 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:09:35.804590 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:09:35.804633 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:09:35.810123 (+ 5490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:35.810994 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:35.811003 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:35.811006 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:35.811153 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:35.811605 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:35.811611 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:35.811617 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:35.812031 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:09:35.812324 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0926 15:09:35.813480 (+ 1156us) server_negotiation.cc:299] Negotiation successful
0926 15:09:35.813738 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23255,"thread_start_us":23037,"threads_started":1}
I20260926 15:09:35.815032 19707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:35.762682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.194:39627 (local address 127.12.34.252:44557)
0926 15:09:35.788574 (+ 25892us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:35.788580 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:35.790023 (+ 1443us) server_negotiation.cc:407] Connection header received
0926 15:09:35.794095 (+ 4072us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:35.794100 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:35.794177 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:35.794252 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:09:35.797018 (+ 2766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:35.798200 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:35.801659 (+ 3459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:35.802398 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:35.802479 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:35.807459 (+ 4980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:09:35.807466 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:09:35.807472 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:09:35.807512 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:09:35.811226 (+ 3714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:35.812092 (+ 866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:35.812097 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:35.812099 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:35.812230 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:35.812713 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:35.812718 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:35.812721 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:35.813048 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:09:35.813207 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0926 15:09:35.814123 (+ 916us) server_negotiation.cc:299] Negotiation successful
0926 15:09:35.814646 (+ 523us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25722,"thread_start_us":25891,"threads_started":1}
I20260926 15:09:35.815378 19039 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:09:35.815981 19713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:35.790649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.194:57379 (local address 127.12.34.254:42537)
0926 15:09:35.793658 (+ 3009us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:35.793666 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:35.793686 (+ 20us) server_negotiation.cc:407] Connection header received
0926 15:09:35.793918 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:35.793923 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:35.794012 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:35.794121 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:09:35.797038 (+ 2917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:35.798138 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:35.801778 (+ 3640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:35.802533 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:35.802839 (+ 306us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:35.808201 (+ 5362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:09:35.808208 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:09:35.808215 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:09:35.808265 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:09:35.811835 (+ 3570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:35.812619 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:35.812625 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:35.812629 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:35.812790 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:35.813171 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:35.813179 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:35.813184 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:35.813507 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:09:35.813649 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0926 15:09:35.815392 (+ 1743us) server_negotiation.cc:299] Negotiation successful
0926 15:09:35.815601 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2864,"thread_start_us":2372,"threads_started":1}
I20260926 15:09:35.816232 19706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:35.762269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.194:59101 (local address 127.12.34.253:32783)
0926 15:09:35.789137 (+ 26868us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:35.789145 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:35.790072 (+ 927us) server_negotiation.cc:407] Connection header received
0926 15:09:35.794021 (+ 3949us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:35.794031 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:35.794123 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:35.794219 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:09:35.797018 (+ 2799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:35.798193 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:35.801689 (+ 3496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:35.802368 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:35.802466 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:35.807802 (+ 5336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:09:35.807807 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:09:35.807812 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:09:35.807848 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:09:35.812129 (+ 4281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:35.813425 (+ 1296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:35.813430 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:35.813434 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:35.813565 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:35.814515 (+ 950us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:35.814520 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:35.814523 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:35.814855 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:09:35.815013 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0926 15:09:35.815504 (+ 491us) server_negotiation.cc:299] Negotiation successful
0926 15:09:35.815693 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26683,"thread_start_us":26533,"threads_started":1}
I20260926 15:09:35.817478 19710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:35.762812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.253:32783 (local address 127.12.34.194:59101)
0926 15:09:35.789822 (+ 27010us) negotiation.cc:107] Waiting for socket to connect
0926 15:09:35.789901 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0926 15:09:35.793616 (+ 3715us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:09:35.794648 (+ 1032us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:09:35.794661 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:09:35.795749 (+ 1088us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:09:35.796800 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:35.796822 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:35.798408 (+ 1586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:35.798412 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:35.801496 (+ 3084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:35.801564 (+ 68us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:35.803540 (+ 1976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:35.803549 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:35.803767 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:35.804261 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:09:35.804284 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:09:35.807598 (+ 3314us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:09:35.812291 (+ 4693us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:09:35.812295 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:09:35.812300 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:09:35.813295 (+ 995us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:09:35.814195 (+ 900us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:09:35.814200 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:09:35.814203 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:09:35.814377 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:09:35.814989 (+ 612us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:09:35.814994 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:09:35.815133 (+ 139us) client_negotiation.cc:769] Sending connection context
0926 15:09:35.815338 (+ 205us) client_negotiation.cc:240] Negotiation successful
0926 15:09:35.815764 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26661,"mutex_wait_us":426,"spinlock_wait_cycles":99072,"thread_start_us":26422,"threads_started":1}
I20260926 15:09:35.818285 19712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:35.790564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:42537 (local address 127.12.34.194:57379)
0926 15:09:35.792935 (+ 2371us) negotiation.cc:107] Waiting for socket to connect
0926 15:09:35.792956 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 15:09:35.793616 (+ 660us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:09:35.794648 (+ 1032us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:09:35.794660 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:09:35.795752 (+ 1092us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:09:35.796827 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:35.796843 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:35.798310 (+ 1467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:35.798317 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:35.801535 (+ 3218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:35.801551 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:35.802896 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:35.802904 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:35.803256 (+ 352us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:35.804442 (+ 1186us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:09:35.804466 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:09:35.807991 (+ 3525us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:09:35.811976 (+ 3985us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:09:35.811982 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:09:35.811986 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:09:35.812494 (+ 508us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:09:35.812890 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:09:35.812895 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:09:35.812899 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:09:35.813035 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:09:35.813689 (+ 654us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:09:35.813694 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:09:35.813820 (+ 126us) client_negotiation.cc:769] Sending connection context
0926 15:09:35.813998 (+ 178us) client_negotiation.cc:240] Negotiation successful
0926 15:09:35.817152 (+ 3154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2243,"mutex_wait_us":919,"spinlock_wait_cycles":17792,"thread_start_us":2519,"threads_started":1}
I20260926 15:09:35.819579 19703 heartbeater.cc:347] Connected to a master server at 127.12.34.254:42537
I20260926 15:09:35.819911 19703 heartbeater.cc:464] Registering TS with master...
I20260926 15:09:35.820819 19703 heartbeater.cc:511] Master 127.12.34.254:42537 requested a full tablet report, sending...
I20260926 15:09:35.821735 19704 heartbeater.cc:347] Connected to a master server at 127.12.34.253:32783
I20260926 15:09:35.821851 19704 heartbeater.cc:464] Registering TS with master...
I20260926 15:09:35.822338 19704 heartbeater.cc:511] Master 127.12.34.253:32783 requested a full tablet report, sending...
I20260926 15:09:35.822535 18587 ts_manager.cc:194] Re-registered known tserver with Master: 85dd0480684b43909497b35c05e43908 (127.12.34.194:38821)
I20260926 15:09:35.822778 19708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:35.762812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.252:44557 (local address 127.12.34.194:39627)
0926 15:09:35.789824 (+ 27012us) negotiation.cc:107] Waiting for socket to connect
0926 15:09:35.789899 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0926 15:09:35.793962 (+ 4063us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:09:35.794930 (+ 968us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:09:35.794935 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:09:35.795749 (+ 814us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:09:35.796811 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:35.796822 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:35.798384 (+ 1562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:35.798391 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:35.801507 (+ 3116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:35.801515 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:35.802897 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:35.802904 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:35.803258 (+ 354us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:35.804156 (+ 898us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:09:35.804187 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:09:35.807168 (+ 2981us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:09:35.811380 (+ 4212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:09:35.811388 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:09:35.811405 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:09:35.811954 (+ 549us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:09:35.812362 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:09:35.812369 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:09:35.812376 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:09:35.812593 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:09:35.813292 (+ 699us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:09:35.813300 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:09:35.813705 (+ 405us) client_negotiation.cc:769] Sending connection context
0926 15:09:35.813998 (+ 293us) client_negotiation.cc:240] Negotiation successful
0926 15:09:35.822316 (+ 8318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26743,"mutex_wait_us":2555,"spinlock_wait_cycles":3968,"thread_start_us":24559,"threads_started":1}
I20260926 15:09:35.823604 19187 ts_manager.cc:194] Registered new tserver with Master: 85dd0480684b43909497b35c05e43908 (127.12.34.194:38821)
I20260926 15:09:35.823984 19702 heartbeater.cc:347] Connected to a master server at 127.12.34.252:44557
I20260926 15:09:35.824055 18587 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.12.34.194@KRBTEST.COM'} at 127.12.34.194:57379
I20260926 15:09:35.824108 19702 heartbeater.cc:464] Registering TS with master...
I20260926 15:09:35.824513 19702 heartbeater.cc:511] Master 127.12.34.252:44557 requested a full tablet report, sending...
I20260926 15:09:35.825582 19375 ts_manager.cc:194] Registered new tserver with Master: 85dd0480684b43909497b35c05e43908 (127.12.34.194:38821)
I20260926 15:09:35.903079 12427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8uoJGv/build/asan/bin/kudu with pid 18916
I20260926 15:09:35.919134 12427 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8uoJGv/build/asan/bin/kudu
/tmp/dist-test-task8uoJGv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.34.195:36967
--local_ip_for_outbound_sockets=127.12.34.195
--tserver_master_addrs=127.12.34.254:42537
--webserver_port=39429
--webserver_interface=127.12.34.195
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/krb5kdc/oryx_127.12.34.195.keytab
--principal=oryx/127.12.34.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.34.212:35299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.12.34.254:42537,127.12.34.253:32783
--tserver_master_addrs=127.12.34.254:42537,127.12.34.253:32783,127.12.34.252:44557 with env {KRB5CCNAME=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 15:09:36.204486 19555 heartbeater.cc:503] Master 127.12.34.254:42537 was elected leader, sending a full tablet report...
W20260926 15:09:36.243760 19715 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:09:36.244095 19715 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:09:36.244261 19715 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:09:36.251534 19715 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:09:36.251678 19715 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:09:36.251850 19715 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.34.195
I20260926 15:09:36.261163 19715 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.34.212:35299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/krb5kdc/oryx_127.12.34.195.keytab
--principal=oryx/127.12.34.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.34.195:36967
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.34.195
--webserver_port=39429
--tserver_master_addrs=127.12.34.254:42537,127.12.34.253:32783,127.12.34.252:44557
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.34.195
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:09:36.262411 19715 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:09:36.263633 19715 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:09:36.276401 19720 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:09:36.276467 19721 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:09:36.276487 19723 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:09:36.277324 19715 server_base.cc:1034] running on GCE node
I20260926 15:09:36.277899 19715 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:09:36.278724 19715 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:09:36.279929 19715 hybrid_clock.cc:648] HybridClock initialized: now 1790435376279900 us; error 30 us; skew 500 ppm
I20260926 15:09:36.280181 19715 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:09:36 dist-test-slave-865v krb5kdc[18543](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435376, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.34.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:09:36.283306 19715 init.cc:377] Logged in from keytab as oryx/127.12.34.195@KRBTEST.COM (short username oryx)
I20260926 15:09:36.284783 19715 webserver.cc:460] Webserver started at http://127.12.34.195:39429/ using document root <none> and password file <none>
I20260926 15:09:36.285450 19715 fs_manager.cc:359] Metadata directory not provided
I20260926 15:09:36.285558 19715 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:09:36.290755 19715 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.002s
I20260926 15:09:36.293304 19730 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:09:36.294749 19715 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 15:09:36.294883 19715 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/minicluster-data/ts-2/data,/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/minicluster-data/ts-2/wal
uuid: "71e96bd442d34587b6d0e98bb1c8aae4"
format_stamp: "Formatted at 2026-09-26 15:09:33 on dist-test-slave-865v"
server_key: "7a2d78a764c744470291a7ee16def260"
server_key_iv: "39a23a6be6f0211c1a793bdfb27717c1"
server_key_version: "encryptionkey@0"
I20260926 15:09:36.295763 19715 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:09:36.319257 19715 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:09:36.323791 19715 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:09:36.324035 19715 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:09:36.324924 19715 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:09:36.326259 19715 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:09:36.326347 19715 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:09:36.326418 19715 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:09:36.326473 19715 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:09:36.360955 19715 rpc_server.cc:307] RPC server started. Bound to: 127.12.34.195:36967
I20260926 15:09:36.361056 19842 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.34.195:36967 every 8 connection(s)
I20260926 15:09:36.362330 19715 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/minicluster-data/ts-2/data/info.pb
I20260926 15:09:36.368614 12427 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8uoJGv/build/asan/bin/kudu as pid 19715
Sep 26 15:09:36 dist-test-slave-865v krb5kdc[18543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435376, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.34.195@KRBTEST.COM for oryx/127.12.34.254@KRBTEST.COM
Sep 26 15:09:36 dist-test-slave-865v krb5kdc[18543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435376, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.34.195@KRBTEST.COM for oryx/127.12.34.253@KRBTEST.COM
Sep 26 15:09:36 dist-test-slave-865v krb5kdc[18543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435376, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.34.195@KRBTEST.COM for oryx/127.12.34.252@KRBTEST.COM
I20260926 15:09:36.390600 19847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:36.365320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.195:37337 (local address 127.12.34.253:32783)
0926 15:09:36.365987 (+ 667us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:36.365994 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:36.367192 (+ 1198us) server_negotiation.cc:407] Connection header received
0926 15:09:36.370089 (+ 2897us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:36.370096 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:36.370224 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:36.370351 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:09:36.373165 (+ 2814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:36.374578 (+ 1413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:36.377178 (+ 2600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:36.378038 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:36.378160 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:36.382439 (+ 4279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:09:36.382445 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:09:36.382449 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:09:36.382486 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:09:36.386000 (+ 3514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:36.386836 (+ 836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:36.386844 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:36.386850 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:36.387002 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:36.387548 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:36.387556 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:36.387562 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:36.387969 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:09:36.388124 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0926 15:09:36.389126 (+ 1002us) server_negotiation.cc:299] Negotiation successful
0926 15:09:36.389340 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":537,"thread_start_us":433,"threads_started":1}
I20260926 15:09:36.390980 19850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:36.366031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.253:32783 (local address 127.12.34.195:37337)
0926 15:09:36.367024 (+ 993us) negotiation.cc:107] Waiting for socket to connect
0926 15:09:36.367087 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0926 15:09:36.369778 (+ 2691us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:09:36.370657 (+ 879us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:09:36.370664 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:09:36.371829 (+ 1165us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:09:36.372932 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:36.372951 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:36.374805 (+ 1854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:36.374813 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:36.376951 (+ 2138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:36.376970 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:36.378544 (+ 1574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:36.378553 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:36.378795 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:36.379430 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:09:36.379462 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:09:36.382148 (+ 2686us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:09:36.386144 (+ 3996us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:09:36.386151 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:09:36.386161 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:09:36.386677 (+ 516us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:09:36.387140 (+ 463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:09:36.387146 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:09:36.387150 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:09:36.387434 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:09:36.388201 (+ 767us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:09:36.388211 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:09:36.388665 (+ 454us) client_negotiation.cc:769] Sending connection context
0926 15:09:36.388982 (+ 317us) client_negotiation.cc:240] Negotiation successful
0926 15:09:36.389623 (+ 641us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":802,"mutex_wait_us":210,"spinlock_wait_cycles":80768,"thread_start_us":449,"threads_started":1}
I20260926 15:09:36.391578 19852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:36.367733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.195:53055 (local address 127.12.34.254:42537)
0926 15:09:36.369398 (+ 1665us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:36.369402 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:36.369417 (+ 15us) server_negotiation.cc:407] Connection header received
0926 15:09:36.370178 (+ 761us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:36.370183 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:36.370259 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:36.370962 (+ 703us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:09:36.373120 (+ 2158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:36.374240 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:36.376088 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:36.376681 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:36.376800 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:36.381595 (+ 4795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:09:36.381602 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:09:36.381606 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:09:36.381646 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:09:36.385941 (+ 4295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:36.387820 (+ 1879us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:36.387827 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:36.387832 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:36.387992 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:36.390086 (+ 2094us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:36.390094 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:36.390097 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:36.390399 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:09:36.390543 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0926 15:09:36.391018 (+ 475us) server_negotiation.cc:299] Negotiation successful
0926 15:09:36.391218 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1572,"thread_start_us":1492,"threads_started":1}
I20260926 15:09:36.393486 19849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:36.366019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.252:44557 (local address 127.12.34.195:60517)
0926 15:09:36.367024 (+ 1005us) negotiation.cc:107] Waiting for socket to connect
0926 15:09:36.367086 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0926 15:09:36.369784 (+ 2698us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:09:36.370621 (+ 837us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:09:36.370632 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:09:36.371801 (+ 1169us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:09:36.372913 (+ 1112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:36.372937 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:36.374421 (+ 1484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:36.374425 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:36.379538 (+ 5113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:36.379555 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:36.381552 (+ 1997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:36.381558 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:36.381748 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:36.382304 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:09:36.382324 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:09:36.385554 (+ 3230us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:09:36.389189 (+ 3635us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:09:36.389194 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:09:36.389197 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:09:36.391455 (+ 2258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:09:36.391893 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:09:36.391896 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:09:36.391898 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:09:36.392011 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:09:36.392531 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:09:36.392536 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:09:36.392670 (+ 134us) client_negotiation.cc:769] Sending connection context
0926 15:09:36.392796 (+ 126us) client_negotiation.cc:240] Negotiation successful
0926 15:09:36.393036 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":706,"mutex_wait_us":202,"spinlock_wait_cycles":52736,"thread_start_us":458,"threads_started":1}
I20260926 15:09:36.393481 19848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:36.365654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.195:60517 (local address 127.12.34.252:44557)
0926 15:09:36.366321 (+ 667us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:36.366329 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:36.367217 (+ 888us) server_negotiation.cc:407] Connection header received
0926 15:09:36.370079 (+ 2862us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:36.370087 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:36.370181 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:36.370259 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:09:36.373117 (+ 2858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:36.374240 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:36.379714 (+ 5474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:36.380235 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:36.380304 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:36.385797 (+ 5493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:09:36.385806 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:09:36.385813 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:09:36.385859 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:09:36.389017 (+ 3158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:36.391588 (+ 2571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:36.391597 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:36.391601 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:36.391744 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:36.392127 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:36.392132 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:36.392136 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:36.392413 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:09:36.392566 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0926 15:09:36.392900 (+ 334us) server_negotiation.cc:299] Negotiation successful
0926 15:09:36.393122 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":539,"thread_start_us":442,"threads_started":1}
I20260926 15:09:36.393635 19845 heartbeater.cc:347] Connected to a master server at 127.12.34.253:32783
I20260926 15:09:36.393949 19845 heartbeater.cc:464] Registering TS with master...
I20260926 15:09:36.394836 19845 heartbeater.cc:511] Master 127.12.34.253:32783 requested a full tablet report, sending...
I20260926 15:09:36.391649 19851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:36.367652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:42537 (local address 127.12.34.195:53055)
0926 15:09:36.368339 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0926 15:09:36.368364 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 15:09:36.369784 (+ 1420us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:09:36.370621 (+ 837us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:09:36.370632 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:09:36.371813 (+ 1181us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:09:36.372915 (+ 1102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:36.372937 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:36.374413 (+ 1476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:36.374418 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:36.375954 (+ 1536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:36.375965 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:36.376885 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:36.376894 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:36.377172 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:36.377892 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:09:36.377921 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:09:36.381271 (+ 3350us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:09:36.386674 (+ 5403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:09:36.386683 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:09:36.386689 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:09:36.387299 (+ 610us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:09:36.389565 (+ 2266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:09:36.389572 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:09:36.389578 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:09:36.389757 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:09:36.390526 (+ 769us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:09:36.390533 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:09:36.390709 (+ 176us) client_negotiation.cc:769] Sending connection context
0926 15:09:36.390874 (+ 165us) client_negotiation.cc:240] Negotiation successful
0926 15:09:36.391183 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"mutex_wait_us":185,"spinlock_wait_cycles":1280,"thread_start_us":345,"threads_started":1}
I20260926 15:09:36.396638 19187 ts_manager.cc:194] Registered new tserver with Master: 71e96bd442d34587b6d0e98bb1c8aae4 (127.12.34.195:36967)
I20260926 15:09:36.396906 19843 heartbeater.cc:347] Connected to a master server at 127.12.34.252:44557
I20260926 15:09:36.397028 19843 heartbeater.cc:464] Registering TS with master...
I20260926 15:09:36.397568 19843 heartbeater.cc:511] Master 127.12.34.252:44557 requested a full tablet report, sending...
I20260926 15:09:36.397895 19844 heartbeater.cc:347] Connected to a master server at 127.12.34.254:42537
I20260926 15:09:36.398039 19844 heartbeater.cc:464] Registering TS with master...
I20260926 15:09:36.398527 19844 heartbeater.cc:511] Master 127.12.34.254:42537 requested a full tablet report, sending...
I20260926 15:09:36.398653 19375 ts_manager.cc:194] Registered new tserver with Master: 71e96bd442d34587b6d0e98bb1c8aae4 (127.12.34.195:36967)
I20260926 15:09:36.399564 18587 ts_manager.cc:194] Re-registered known tserver with Master: 71e96bd442d34587b6d0e98bb1c8aae4 (127.12.34.195:36967)
I20260926 15:09:36.401412 18587 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.12.34.195@KRBTEST.COM'} at 127.12.34.195:53055
I20260926 15:09:36.418174 19852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:36.383875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36816 (local address 127.12.34.254:42537)
0926 15:09:36.391817 (+ 7942us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:36.391822 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:36.402664 (+ 10842us) server_negotiation.cc:407] Connection header received
0926 15:09:36.403194 (+ 530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:36.403203 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:36.403330 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:36.403440 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:09:36.405316 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:36.406020 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:36.407445 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:36.407831 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:36.407964 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:36.412210 (+ 4246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:09:36.412215 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:09:36.412218 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:09:36.412247 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:09:36.415472 (+ 3225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:36.416184 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:36.416190 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:36.416195 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:36.416345 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:36.416760 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:36.416765 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:36.416769 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:36.417031 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:09:36.417285 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0926 15:09:36.417699 (+ 414us) server_negotiation.cc:299] Negotiation successful
0926 15:09:36.417920 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7828,"thread_start_us":18261,"threads_started":1}
I20260926 15:09:36.421792 19847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:36.401556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44540 (local address 127.12.34.253:32783)
0926 15:09:36.401734 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:36.401741 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:36.404897 (+ 3156us) server_negotiation.cc:407] Connection header received
0926 15:09:36.405297 (+ 400us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:36.405364 (+ 67us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:36.405465 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:36.405553 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:09:36.408891 (+ 3338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:36.409673 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:36.411097 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:36.411894 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:36.412013 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:36.415454 (+ 3441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:09:36.415459 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:09:36.415462 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:09:36.415525 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:09:36.418979 (+ 3454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:36.419740 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:36.419745 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:36.419748 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:36.419892 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:36.420356 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:36.420362 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:36.420368 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:36.420700 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:09:36.420947 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0926 15:09:36.421283 (+ 336us) server_negotiation.cc:299] Negotiation successful
0926 15:09:36.421488 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 15:09:36.425205 19848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:36.402878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54118 (local address 127.12.34.252:44557)
0926 15:09:36.403099 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:36.403107 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:36.408632 (+ 5525us) server_negotiation.cc:407] Connection header received
0926 15:09:36.408824 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:36.408829 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:36.408896 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:36.408997 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:09:36.411269 (+ 2272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:36.412262 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:36.414092 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:36.414750 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:36.414874 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:36.418672 (+ 3798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:09:36.418678 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:09:36.418682 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:09:36.418722 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:09:36.421808 (+ 3086us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:36.422594 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:36.422599 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:36.422602 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:36.422742 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:36.423828 (+ 1086us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:36.423834 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:36.423837 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:36.424094 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:09:36.424265 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0926 15:09:36.424667 (+ 402us) server_negotiation.cc:299] Negotiation successful
0926 15:09:36.424845 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 15:09:36.423038 18587 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36816:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 15:09:36.430888 18587 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 15:09:36.460001 19870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:36.448113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.194:38821 (local address 127.0.0.1:40464)
0926 15:09:36.449161 (+ 1048us) negotiation.cc:107] Waiting for socket to connect
0926 15:09:36.449181 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 15:09:36.449314 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:09:36.450214 (+ 900us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:09:36.450221 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:09:36.450248 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:09:36.450617 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:36.450631 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:36.452757 (+ 2126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:36.452765 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:36.455672 (+ 2907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:36.455691 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:36.458741 (+ 3050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:36.458751 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:36.458940 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:36.458971 (+ 31us) client_negotiation.cc:769] Sending connection context
0926 15:09:36.459062 (+ 91us) client_negotiation.cc:240] Negotiation successful
0926 15:09:36.459450 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":904,"thread_start_us":1430,"threads_started":1}
I20260926 15:09:36.462139 19869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:36.448336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40464 (local address 127.12.34.194:38821)
0926 15:09:36.449111 (+ 775us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:36.449122 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:36.449276 (+ 154us) server_negotiation.cc:407] Connection header received
0926 15:09:36.449483 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:36.449491 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:36.449796 (+ 305us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:36.449988 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:09:36.450820 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:36.452563 (+ 1743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:36.455888 (+ 3325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:36.457714 (+ 1826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:36.457872 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:36.458118 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0926 15:09:36.461480 (+ 3362us) server_negotiation.cc:299] Negotiation successful
0926 15:09:36.461728 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":591,"thread_start_us":502,"threads_started":1}
I20260926 15:09:36.471024 19637 tablet_service.cc:1467] Processing CreateTablet for tablet 07023488e9454f0e9edc8faedd076ec8 (DEFAULT_TABLE table=test-table [id=745cd4d673724f5ab276c3dfffb91f76]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:09:36.472380 19637 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07023488e9454f0e9edc8faedd076ec8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:09:36.475785 19870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:36.449853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.193:38979 (local address 127.0.0.1:57010)
0926 15:09:36.460234 (+ 10381us) negotiation.cc:107] Waiting for socket to connect
0926 15:09:36.460251 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 15:09:36.460348 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:09:36.462425 (+ 2077us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:09:36.462433 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:09:36.462470 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:09:36.462725 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:36.462738 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:36.468841 (+ 6103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:36.468847 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:36.473036 (+ 4189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:36.473047 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:36.475116 (+ 2069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:36.475120 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:36.475235 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:36.475257 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 15:09:36.475358 (+ 101us) client_negotiation.cc:240] Negotiation successful
0926 15:09:36.475490 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10297,"thread_start_us":13594,"threads_started":1}
I20260926 15:09:36.475829 19874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:36.450368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57010 (local address 127.12.34.193:38979)
0926 15:09:36.458535 (+ 8167us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:36.458547 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:36.460492 (+ 1945us) server_negotiation.cc:407] Connection header received
0926 15:09:36.460613 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:36.460624 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:36.460957 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:36.461208 (+ 251us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:09:36.463172 (+ 1964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:36.468222 (+ 5050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:36.473259 (+ 5037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:36.474696 (+ 1437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:36.474833 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:36.475036 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0926 15:09:36.475410 (+ 374us) server_negotiation.cc:299] Negotiation successful
0926 15:09:36.475553 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7981,"thread_start_us":7880,"threads_started":1}
I20260926 15:09:36.479610 19489 tablet_service.cc:1467] Processing CreateTablet for tablet 07023488e9454f0e9edc8faedd076ec8 (DEFAULT_TABLE table=test-table [id=745cd4d673724f5ab276c3dfffb91f76]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:09:36.480665 19489 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07023488e9454f0e9edc8faedd076ec8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:09:36.490351 19871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:36.449881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.195:36967 (local address 127.0.0.1:38924)
0926 15:09:36.463231 (+ 13350us) negotiation.cc:107] Waiting for socket to connect
0926 15:09:36.463252 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 15:09:36.463366 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:09:36.471647 (+ 8281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:09:36.471655 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:09:36.471683 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:09:36.472095 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:36.472106 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:36.485154 (+ 13048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:36.485160 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:36.487198 (+ 2038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:36.487230 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:36.489244 (+ 2014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:36.489253 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:36.489518 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:36.489572 (+ 54us) client_negotiation.cc:769] Sending connection context
0926 15:09:36.489673 (+ 101us) client_negotiation.cc:240] Negotiation successful
0926 15:09:36.489876 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13222,"thread_start_us":14708,"threads_started":1}
I20260926 15:09:36.491460 19873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:36.450206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38924 (local address 127.12.34.195:36967)
0926 15:09:36.471093 (+ 20887us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:36.471098 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:36.471123 (+ 25us) server_negotiation.cc:407] Connection header received
0926 15:09:36.471192 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:36.471197 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:36.471447 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:36.471634 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:09:36.473217 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:36.474870 (+ 1653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:36.487394 (+ 12524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:36.488997 (+ 1603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:36.489147 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:36.490867 (+ 1720us) server_negotiation.cc:1035] Waiting for connection context
0926 15:09:36.490942 (+ 75us) server_negotiation.cc:299] Negotiation successful
0926 15:09:36.491103 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20096,"thread_start_us":20164,"threads_started":1}
I20260926 15:09:36.495311 19875 tablet_bootstrap.cc:492] T 07023488e9454f0e9edc8faedd076ec8 P 85dd0480684b43909497b35c05e43908: Bootstrap starting.
I20260926 15:09:36.495975 19778 tablet_service.cc:1467] Processing CreateTablet for tablet 07023488e9454f0e9edc8faedd076ec8 (DEFAULT_TABLE table=test-table [id=745cd4d673724f5ab276c3dfffb91f76]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:09:36.497767 19778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07023488e9454f0e9edc8faedd076ec8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:09:36.499343 19875 tablet_bootstrap.cc:654] T 07023488e9454f0e9edc8faedd076ec8 P 85dd0480684b43909497b35c05e43908: Neither blocks nor log segments found. Creating new log.
I20260926 15:09:36.500630 19875 log.cc:824] T 07023488e9454f0e9edc8faedd076ec8 P 85dd0480684b43909497b35c05e43908: Log is configured to *not* fsync() on all Append() calls
I20260926 15:09:36.503362 19876 tablet_bootstrap.cc:492] T 07023488e9454f0e9edc8faedd076ec8 P d3e156f17705476c90ae1f60578daa75: Bootstrap starting.
I20260926 15:09:36.507257 19876 tablet_bootstrap.cc:654] T 07023488e9454f0e9edc8faedd076ec8 P d3e156f17705476c90ae1f60578daa75: Neither blocks nor log segments found. Creating new log.
I20260926 15:09:36.508544 19876 log.cc:824] T 07023488e9454f0e9edc8faedd076ec8 P d3e156f17705476c90ae1f60578daa75: Log is configured to *not* fsync() on all Append() calls
I20260926 15:09:36.513902 19875 tablet_bootstrap.cc:492] T 07023488e9454f0e9edc8faedd076ec8 P 85dd0480684b43909497b35c05e43908: No bootstrap required, opened a new log
I20260926 15:09:36.514173 19875 ts_tablet_manager.cc:1397] T 07023488e9454f0e9edc8faedd076ec8 P 85dd0480684b43909497b35c05e43908: Time spent bootstrapping tablet: real 0.019s user 0.008s sys 0.005s
I20260926 15:09:36.524104 19876 tablet_bootstrap.cc:492] T 07023488e9454f0e9edc8faedd076ec8 P d3e156f17705476c90ae1f60578daa75: No bootstrap required, opened a new log
I20260926 15:09:36.524415 19876 ts_tablet_manager.cc:1397] T 07023488e9454f0e9edc8faedd076ec8 P d3e156f17705476c90ae1f60578daa75: Time spent bootstrapping tablet: real 0.021s user 0.003s sys 0.015s
I20260926 15:09:36.526561 19879 tablet_bootstrap.cc:492] T 07023488e9454f0e9edc8faedd076ec8 P 71e96bd442d34587b6d0e98bb1c8aae4: Bootstrap starting.
I20260926 15:09:36.526337 19875 raft_consensus.cc:348] T 07023488e9454f0e9edc8faedd076ec8 P 85dd0480684b43909497b35c05e43908 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85dd0480684b43909497b35c05e43908" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 38821 } } peers { permanent_uuid: "d3e156f17705476c90ae1f60578daa75" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 38979 } } peers { permanent_uuid: "71e96bd442d34587b6d0e98bb1c8aae4" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 36967 } }
I20260926 15:09:36.526767 19875 raft_consensus.cc:374] T 07023488e9454f0e9edc8faedd076ec8 P 85dd0480684b43909497b35c05e43908 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:09:36.526873 19875 raft_consensus.cc:729] T 07023488e9454f0e9edc8faedd076ec8 P 85dd0480684b43909497b35c05e43908 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85dd0480684b43909497b35c05e43908, State: Initialized, Role: FOLLOWER
I20260926 15:09:36.527614 19875 consensus_queue.cc:260] T 07023488e9454f0e9edc8faedd076ec8 P 85dd0480684b43909497b35c05e43908 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85dd0480684b43909497b35c05e43908" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 38821 } } peers { permanent_uuid: "d3e156f17705476c90ae1f60578daa75" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 38979 } } peers { permanent_uuid: "71e96bd442d34587b6d0e98bb1c8aae4" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 36967 } }
I20260926 15:09:36.531289 19879 tablet_bootstrap.cc:654] T 07023488e9454f0e9edc8faedd076ec8 P 71e96bd442d34587b6d0e98bb1c8aae4: Neither blocks nor log segments found. Creating new log.
I20260926 15:09:36.533360 19879 log.cc:824] T 07023488e9454f0e9edc8faedd076ec8 P 71e96bd442d34587b6d0e98bb1c8aae4: Log is configured to *not* fsync() on all Append() calls
I20260926 15:09:36.535892 19875 ts_tablet_manager.cc:1428] T 07023488e9454f0e9edc8faedd076ec8 P 85dd0480684b43909497b35c05e43908: Time spent starting tablet: real 0.022s user 0.015s sys 0.006s
I20260926 15:09:36.538105 19876 raft_consensus.cc:348] T 07023488e9454f0e9edc8faedd076ec8 P d3e156f17705476c90ae1f60578daa75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85dd0480684b43909497b35c05e43908" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 38821 } } peers { permanent_uuid: "d3e156f17705476c90ae1f60578daa75" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 38979 } } peers { permanent_uuid: "71e96bd442d34587b6d0e98bb1c8aae4" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 36967 } }
I20260926 15:09:36.538564 19876 raft_consensus.cc:374] T 07023488e9454f0e9edc8faedd076ec8 P d3e156f17705476c90ae1f60578daa75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:09:36.538657 19876 raft_consensus.cc:729] T 07023488e9454f0e9edc8faedd076ec8 P d3e156f17705476c90ae1f60578daa75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3e156f17705476c90ae1f60578daa75, State: Initialized, Role: FOLLOWER
I20260926 15:09:36.539394 19876 consensus_queue.cc:260] T 07023488e9454f0e9edc8faedd076ec8 P d3e156f17705476c90ae1f60578daa75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85dd0480684b43909497b35c05e43908" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 38821 } } peers { permanent_uuid: "d3e156f17705476c90ae1f60578daa75" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 38979 } } peers { permanent_uuid: "71e96bd442d34587b6d0e98bb1c8aae4" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 36967 } }
I20260926 15:09:36.540059 19703 heartbeater.cc:503] Master 127.12.34.254:42537 was elected leader, sending a full tablet report...
I20260926 15:09:36.552790 19879 tablet_bootstrap.cc:492] T 07023488e9454f0e9edc8faedd076ec8 P 71e96bd442d34587b6d0e98bb1c8aae4: No bootstrap required, opened a new log
I20260926 15:09:36.553160 19879 ts_tablet_manager.cc:1397] T 07023488e9454f0e9edc8faedd076ec8 P 71e96bd442d34587b6d0e98bb1c8aae4: Time spent bootstrapping tablet: real 0.027s user 0.010s sys 0.012s
I20260926 15:09:36.558502 19876 ts_tablet_manager.cc:1428] T 07023488e9454f0e9edc8faedd076ec8 P d3e156f17705476c90ae1f60578daa75: Time spent starting tablet: real 0.034s user 0.017s sys 0.015s
I20260926 15:09:36.563611 19879 raft_consensus.cc:348] T 07023488e9454f0e9edc8faedd076ec8 P 71e96bd442d34587b6d0e98bb1c8aae4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85dd0480684b43909497b35c05e43908" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 38821 } } peers { permanent_uuid: "d3e156f17705476c90ae1f60578daa75" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 38979 } } peers { permanent_uuid: "71e96bd442d34587b6d0e98bb1c8aae4" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 36967 } }
I20260926 15:09:36.563992 19879 raft_consensus.cc:374] T 07023488e9454f0e9edc8faedd076ec8 P 71e96bd442d34587b6d0e98bb1c8aae4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:09:36.564116 19879 raft_consensus.cc:729] T 07023488e9454f0e9edc8faedd076ec8 P 71e96bd442d34587b6d0e98bb1c8aae4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71e96bd442d34587b6d0e98bb1c8aae4, State: Initialized, Role: FOLLOWER
I20260926 15:09:36.564669 19879 consensus_queue.cc:260] T 07023488e9454f0e9edc8faedd076ec8 P 71e96bd442d34587b6d0e98bb1c8aae4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85dd0480684b43909497b35c05e43908" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 38821 } } peers { permanent_uuid: "d3e156f17705476c90ae1f60578daa75" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 38979 } } peers { permanent_uuid: "71e96bd442d34587b6d0e98bb1c8aae4" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 36967 } }
I20260926 15:09:36.565781 19844 heartbeater.cc:503] Master 127.12.34.254:42537 was elected leader, sending a full tablet report...
I20260926 15:09:36.566500 19879 ts_tablet_manager.cc:1428] T 07023488e9454f0e9edc8faedd076ec8 P 71e96bd442d34587b6d0e98bb1c8aae4: Time spent starting tablet: real 0.013s user 0.007s sys 0.005s
I20260926 15:09:36.583947 19880 raft_consensus.cc:482] T 07023488e9454f0e9edc8faedd076ec8 P 85dd0480684b43909497b35c05e43908 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 15:09:36.584281 19880 raft_consensus.cc:504] T 07023488e9454f0e9edc8faedd076ec8 P 85dd0480684b43909497b35c05e43908 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85dd0480684b43909497b35c05e43908" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 38821 } } peers { permanent_uuid: "d3e156f17705476c90ae1f60578daa75" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 38979 } } peers { permanent_uuid: "71e96bd442d34587b6d0e98bb1c8aae4" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 36967 } }
I20260926 15:09:36.586509 19880 leader_election.cc:288] T 07023488e9454f0e9edc8faedd076ec8 P 85dd0480684b43909497b35c05e43908 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3e156f17705476c90ae1f60578daa75 (127.12.34.193:38979), 71e96bd442d34587b6d0e98bb1c8aae4 (127.12.34.195:36967)
I20260926 15:09:36.599607 19874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:36.587280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.194:48697 (local address 127.12.34.193:38979)
0926 15:09:36.587513 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:36.587521 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:36.591229 (+ 3708us) server_negotiation.cc:407] Connection header received
0926 15:09:36.591896 (+ 667us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:36.591901 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:36.591970 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:36.592051 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:09:36.593277 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:36.594330 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:36.597026 (+ 2696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:36.598285 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:36.598405 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:36.598471 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 15:09:36.599229 (+ 758us) server_negotiation.cc:299] Negotiation successful
0926 15:09:36.599364 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 15:09:36.599611 19873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:36.586864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.194:38803 (local address 127.12.34.195:36967)
0926 15:09:36.587073 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:36.587082 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:36.591470 (+ 4388us) server_negotiation.cc:407] Connection header received
0926 15:09:36.591747 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:36.591753 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:36.591829 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:36.591909 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:09:36.593222 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:36.594333 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:36.597037 (+ 2704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:36.598285 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:36.598402 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:36.598471 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 15:09:36.599238 (+ 767us) server_negotiation.cc:299] Negotiation successful
0926 15:09:36.599364 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 15:09:36.600301 19884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:36.587565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.195:36967 (local address 127.12.34.194:38803)
0926 15:09:36.591363 (+ 3798us) negotiation.cc:107] Waiting for socket to connect
0926 15:09:36.591377 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 15:09:36.591570 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:09:36.592069 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:09:36.592073 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:09:36.592095 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:09:36.592975 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:36.592990 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:36.594506 (+ 1516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:36.594511 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:36.596863 (+ 2352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:36.596871 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:36.598662 (+ 1791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:36.598674 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:36.598995 (+ 321us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:36.599079 (+ 84us) client_negotiation.cc:769] Sending connection context
0926 15:09:36.599797 (+ 718us) client_negotiation.cc:240] Negotiation successful
0926 15:09:36.599945 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3698,"mutex_wait_us":247,"thread_start_us":3106,"threads_started":1}
I20260926 15:09:36.601441 19798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07023488e9454f0e9edc8faedd076ec8" candidate_uuid: "85dd0480684b43909497b35c05e43908" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71e96bd442d34587b6d0e98bb1c8aae4" is_pre_election: true
I20260926 15:09:36.601927 19798 raft_consensus.cc:2454] T 07023488e9454f0e9edc8faedd076ec8 P 71e96bd442d34587b6d0e98bb1c8aae4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85dd0480684b43909497b35c05e43908 in term 0.
I20260926 15:09:36.602643 19885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:36.586833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.193:38979 (local address 127.12.34.194:48697)
0926 15:09:36.591108 (+ 4275us) negotiation.cc:107] Waiting for socket to connect
0926 15:09:36.591132 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 15:09:36.591747 (+ 615us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:09:36.592683 (+ 936us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:09:36.592690 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:09:36.592717 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:09:36.593115 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:36.593129 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:36.594506 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:36.594511 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:36.596857 (+ 2346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:36.596871 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:36.598685 (+ 1814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:36.598690 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:36.599039 (+ 349us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:36.599082 (+ 43us) client_negotiation.cc:769] Sending connection context
0926 15:09:36.600777 (+ 1695us) client_negotiation.cc:240] Negotiation successful
0926 15:09:36.600927 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4094,"mutex_wait_us":178,"thread_start_us":4280,"threads_started":1}
I20260926 15:09:36.602666 19590 leader_election.cc:302] T 07023488e9454f0e9edc8faedd076ec8 P 85dd0480684b43909497b35c05e43908 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71e96bd442d34587b6d0e98bb1c8aae4, 85dd0480684b43909497b35c05e43908; no voters:
I20260926 15:09:36.603123 19880 raft_consensus.cc:2781] T 07023488e9454f0e9edc8faedd076ec8 P 85dd0480684b43909497b35c05e43908 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 15:09:36.603309 19880 raft_consensus.cc:482] T 07023488e9454f0e9edc8faedd076ec8 P 85dd0480684b43909497b35c05e43908 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 15:09:36.603434 19880 raft_consensus.cc:3037] T 07023488e9454f0e9edc8faedd076ec8 P 85dd0480684b43909497b35c05e43908 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:09:36.603546 19509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07023488e9454f0e9edc8faedd076ec8" candidate_uuid: "85dd0480684b43909497b35c05e43908" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3e156f17705476c90ae1f60578daa75" is_pre_election: true
I20260926 15:09:36.603967 19509 raft_consensus.cc:2454] T 07023488e9454f0e9edc8faedd076ec8 P d3e156f17705476c90ae1f60578daa75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85dd0480684b43909497b35c05e43908 in term 0.
I20260926 15:09:36.607308 19880 raft_consensus.cc:504] T 07023488e9454f0e9edc8faedd076ec8 P 85dd0480684b43909497b35c05e43908 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85dd0480684b43909497b35c05e43908" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 38821 } } peers { permanent_uuid: "d3e156f17705476c90ae1f60578daa75" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 38979 } } peers { permanent_uuid: "71e96bd442d34587b6d0e98bb1c8aae4" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 36967 } }
I20260926 15:09:36.608073 19880 leader_election.cc:288] T 07023488e9454f0e9edc8faedd076ec8 P 85dd0480684b43909497b35c05e43908 [CANDIDATE]: Term 1 election: Requested vote from peers d3e156f17705476c90ae1f60578daa75 (127.12.34.193:38979), 71e96bd442d34587b6d0e98bb1c8aae4 (127.12.34.195:36967)
I20260926 15:09:36.608472 19509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07023488e9454f0e9edc8faedd076ec8" candidate_uuid: "85dd0480684b43909497b35c05e43908" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3e156f17705476c90ae1f60578daa75"
I20260926 15:09:36.608489 19798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07023488e9454f0e9edc8faedd076ec8" candidate_uuid: "85dd0480684b43909497b35c05e43908" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71e96bd442d34587b6d0e98bb1c8aae4"
I20260926 15:09:36.608651 19509 raft_consensus.cc:3037] T 07023488e9454f0e9edc8faedd076ec8 P d3e156f17705476c90ae1f60578daa75 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:09:36.608661 19798 raft_consensus.cc:3037] T 07023488e9454f0e9edc8faedd076ec8 P 71e96bd442d34587b6d0e98bb1c8aae4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:09:36.612208 19798 raft_consensus.cc:2454] T 07023488e9454f0e9edc8faedd076ec8 P 71e96bd442d34587b6d0e98bb1c8aae4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85dd0480684b43909497b35c05e43908 in term 1.
I20260926 15:09:36.612571 19590 leader_election.cc:302] T 07023488e9454f0e9edc8faedd076ec8 P 85dd0480684b43909497b35c05e43908 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71e96bd442d34587b6d0e98bb1c8aae4, 85dd0480684b43909497b35c05e43908; no voters:
I20260926 15:09:36.612862 19880 raft_consensus.cc:2781] T 07023488e9454f0e9edc8faedd076ec8 P 85dd0480684b43909497b35c05e43908 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:09:36.613148 19509 raft_consensus.cc:2454] T 07023488e9454f0e9edc8faedd076ec8 P d3e156f17705476c90ae1f60578daa75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85dd0480684b43909497b35c05e43908 in term 1.
I20260926 15:09:36.614044 19880 raft_consensus.cc:686] T 07023488e9454f0e9edc8faedd076ec8 P 85dd0480684b43909497b35c05e43908 [term 1 LEADER]: Becoming Leader. State: Replica: 85dd0480684b43909497b35c05e43908, State: Running, Role: LEADER
I20260926 15:09:36.614549 19880 consensus_queue.cc:237] T 07023488e9454f0e9edc8faedd076ec8 P 85dd0480684b43909497b35c05e43908 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85dd0480684b43909497b35c05e43908" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 38821 } } peers { permanent_uuid: "d3e156f17705476c90ae1f60578daa75" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 38979 } } peers { permanent_uuid: "71e96bd442d34587b6d0e98bb1c8aae4" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 36967 } }
W20260926 15:09:36.615926 19846 tablet.cc:2369] T 07023488e9454f0e9edc8faedd076ec8 P 71e96bd442d34587b6d0e98bb1c8aae4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 15:09:36.619547 18587 catalog_manager.cc:5495] T 07023488e9454f0e9edc8faedd076ec8 P 85dd0480684b43909497b35c05e43908 reported cstate change: term changed from 0 to 1, leader changed from <none> to 85dd0480684b43909497b35c05e43908 (127.12.34.194). New cstate: current_term: 1 leader_uuid: "85dd0480684b43909497b35c05e43908" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85dd0480684b43909497b35c05e43908" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 38821 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3e156f17705476c90ae1f60578daa75" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 38979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "71e96bd442d34587b6d0e98bb1c8aae4" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 36967 } health_report { overall_health: UNKNOWN } } }
W20260926 15:09:36.653865 19557 tablet.cc:2369] T 07023488e9454f0e9edc8faedd076ec8 P d3e156f17705476c90ae1f60578daa75: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 15:09:36.662559 19869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:36.657336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40466 (local address 127.12.34.194:38821)
0926 15:09:36.657698 (+ 362us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:36.657705 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:36.657727 (+ 22us) server_negotiation.cc:407] Connection header received
0926 15:09:36.657940 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:36.657944 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:36.658015 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:36.658159 (+ 144us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 15:09:36.658496 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:36.659204 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:36.660490 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:36.660898 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:36.661032 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:36.661538 (+ 506us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 15:09:36.662017 (+ 479us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 15:09:36.662175 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0926 15:09:36.662230 (+ 55us) server_negotiation.cc:299] Negotiation successful
0926 15:09:36.662306 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":224}
I20260926 15:09:36.670655 19509 raft_consensus.cc:1261] T 07023488e9454f0e9edc8faedd076ec8 P d3e156f17705476c90ae1f60578daa75 [term 1 FOLLOWER]: Refusing update from remote peer 85dd0480684b43909497b35c05e43908: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:09:36.671241 19798 raft_consensus.cc:1261] T 07023488e9454f0e9edc8faedd076ec8 P 71e96bd442d34587b6d0e98bb1c8aae4 [term 1 FOLLOWER]: Refusing update from remote peer 85dd0480684b43909497b35c05e43908: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:09:36.671777 19886 consensus_queue.cc:1035] T 07023488e9454f0e9edc8faedd076ec8 P 85dd0480684b43909497b35c05e43908 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3e156f17705476c90ae1f60578daa75" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 38979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:09:36.672221 19880 consensus_queue.cc:1035] T 07023488e9454f0e9edc8faedd076ec8 P 85dd0480684b43909497b35c05e43908 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71e96bd442d34587b6d0e98bb1c8aae4" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 36967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:09:36.711400 18587 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36816:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 15:09:36.711651 18587 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36816:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 15:09:36.721238 18587 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f8c077ce1f004166989faaf5668ef1e8: Sending DeleteTablet for 3 replicas of tablet 07023488e9454f0e9edc8faedd076ec8
I20260926 15:09:36.722494 19637 tablet_service.cc:1514] Processing DeleteTablet for tablet 07023488e9454f0e9edc8faedd076ec8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:09:36 UTC) from {username='oryx', principal='oryx/127.12.34.254@KRBTEST.COM'} at 127.0.0.1:40464
I20260926 15:09:36.723277 19778 tablet_service.cc:1514] Processing DeleteTablet for tablet 07023488e9454f0e9edc8faedd076ec8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:09:36 UTC) from {username='oryx', principal='oryx/127.12.34.254@KRBTEST.COM'} at 127.0.0.1:38924
I20260926 15:09:36.729529 19900 tablet_replica.cc:330] T 07023488e9454f0e9edc8faedd076ec8 P 71e96bd442d34587b6d0e98bb1c8aae4: stopping tablet replica
I20260926 15:09:36.729972 19900 raft_consensus.cc:2229] T 07023488e9454f0e9edc8faedd076ec8 P 71e96bd442d34587b6d0e98bb1c8aae4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 15:09:36.730453 19900 raft_consensus.cc:2258] T 07023488e9454f0e9edc8faedd076ec8 P 71e96bd442d34587b6d0e98bb1c8aae4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:09:36.733421 19900 ts_tablet_manager.cc:1905] T 07023488e9454f0e9edc8faedd076ec8 P 71e96bd442d34587b6d0e98bb1c8aae4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:09:36.734174 19489 tablet_service.cc:1514] Processing DeleteTablet for tablet 07023488e9454f0e9edc8faedd076ec8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:09:36 UTC) from {username='oryx', principal='oryx/127.12.34.254@KRBTEST.COM'} at 127.0.0.1:57010
I20260926 15:09:36.734556 19778 tablet_service.cc:1514] Processing DeleteTablet for tablet 07023488e9454f0e9edc8faedd076ec8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:09:36 UTC) from {username='oryx', principal='oryx/127.12.34.254@KRBTEST.COM'} at 127.0.0.1:38924
I20260926 15:09:36.741401 19900 ts_tablet_manager.cc:1918] T 07023488e9454f0e9edc8faedd076ec8 P 71e96bd442d34587b6d0e98bb1c8aae4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 15:09:36.741618 19900 log.cc:1196] T 07023488e9454f0e9edc8faedd076ec8 P 71e96bd442d34587b6d0e98bb1c8aae4: Deleting WAL directory at /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/minicluster-data/ts-2/wal/wals/07023488e9454f0e9edc8faedd076ec8
I20260926 15:09:36.742297 19900 ts_tablet_manager.cc:1939] T 07023488e9454f0e9edc8faedd076ec8 P 71e96bd442d34587b6d0e98bb1c8aae4: Deleting consensus metadata
I20260926 15:09:36.744205 18573 catalog_manager.cc:4841] TS 71e96bd442d34587b6d0e98bb1c8aae4 (127.12.34.195:36967): tablet 07023488e9454f0e9edc8faedd076ec8 (table test-table [id=745cd4d673724f5ab276c3dfffb91f76]) successfully deleted
W20260926 15:09:36.744807 18573 catalog_manager.cc:4808] TS 71e96bd442d34587b6d0e98bb1c8aae4 (127.12.34.195:36967): delete failed for tablet 07023488e9454f0e9edc8faedd076ec8 because the tablet was not found. No further retry: Not found: Tablet not found: 07023488e9454f0e9edc8faedd076ec8
I20260926 15:09:36.748416 19899 tablet_replica.cc:330] T 07023488e9454f0e9edc8faedd076ec8 P 85dd0480684b43909497b35c05e43908: stopping tablet replica
I20260926 15:09:36.748885 19899 raft_consensus.cc:2229] T 07023488e9454f0e9edc8faedd076ec8 P 85dd0480684b43909497b35c05e43908 [term 1 LEADER]: Raft consensus shutting down.
I20260926 15:09:36.749581 19899 raft_consensus.cc:2258] T 07023488e9454f0e9edc8faedd076ec8 P 85dd0480684b43909497b35c05e43908 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:09:36.749855 12427 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 18556
I20260926 15:09:36.750330 19901 tablet_replica.cc:330] T 07023488e9454f0e9edc8faedd076ec8 P d3e156f17705476c90ae1f60578daa75: stopping tablet replica
I20260926 15:09:36.750707 19901 raft_consensus.cc:2229] T 07023488e9454f0e9edc8faedd076ec8 P d3e156f17705476c90ae1f60578daa75 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 15:09:36.750906 18618 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:09:36.751253 19901 raft_consensus.cc:2258] T 07023488e9454f0e9edc8faedd076ec8 P d3e156f17705476c90ae1f60578daa75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:09:36.753719 19899 ts_tablet_manager.cc:1905] T 07023488e9454f0e9edc8faedd076ec8 P 85dd0480684b43909497b35c05e43908: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:09:36.759784 19901 ts_tablet_manager.cc:1905] T 07023488e9454f0e9edc8faedd076ec8 P d3e156f17705476c90ae1f60578daa75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:09:36.781314 19847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:36.738700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44552 (local address 127.12.34.253:32783)
0926 15:09:36.738902 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:36.738909 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:36.748210 (+ 9301us) server_negotiation.cc:407] Connection header received
0926 15:09:36.749198 (+ 988us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:36.749205 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:36.749285 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:36.749358 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:09:36.757202 (+ 7844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:36.758304 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:36.760230 (+ 1926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:36.760875 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:36.760971 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:36.765253 (+ 4282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:09:36.765260 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:09:36.765266 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:09:36.765305 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:09:36.768667 (+ 3362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:36.777521 (+ 8854us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:36.777528 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:36.777532 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:36.777710 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:36.778187 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:36.778192 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:36.778195 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:36.778614 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:09:36.779095 (+ 481us) server_negotiation.cc:1035] Waiting for connection context
0926 15:09:36.780655 (+ 1560us) server_negotiation.cc:299] Negotiation successful
0926 15:09:36.780937 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 15:09:36.781450 19899 ts_tablet_manager.cc:1918] T 07023488e9454f0e9edc8faedd076ec8 P 85dd0480684b43909497b35c05e43908: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 15:09:36.781622 19899 log.cc:1196] T 07023488e9454f0e9edc8faedd076ec8 P 85dd0480684b43909497b35c05e43908: Deleting WAL directory at /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/minicluster-data/ts-1/wal/wals/07023488e9454f0e9edc8faedd076ec8
I20260926 15:09:36.782137 19899 ts_tablet_manager.cc:1939] T 07023488e9454f0e9edc8faedd076ec8 P 85dd0480684b43909497b35c05e43908: Deleting consensus metadata
I20260926 15:09:36.788760 19901 ts_tablet_manager.cc:1918] T 07023488e9454f0e9edc8faedd076ec8 P d3e156f17705476c90ae1f60578daa75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 15:09:36.788924 19901 log.cc:1196] T 07023488e9454f0e9edc8faedd076ec8 P d3e156f17705476c90ae1f60578daa75: Deleting WAL directory at /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/minicluster-data/ts-0/wal/wals/07023488e9454f0e9edc8faedd076ec8
I20260926 15:09:36.789456 19901 ts_tablet_manager.cc:1939] T 07023488e9454f0e9edc8faedd076ec8 P d3e156f17705476c90ae1f60578daa75: Deleting consensus metadata
I20260926 15:09:36.804989 19848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:36.748442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54120 (local address 127.12.34.252:44557)
0926 15:09:36.748660 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:36.748669 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:36.782816 (+ 34147us) server_negotiation.cc:407] Connection header received
0926 15:09:36.782918 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:36.782926 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:36.783019 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:36.783120 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:09:36.786476 (+ 3356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:36.787858 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:36.790467 (+ 2609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:36.791309 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:36.791420 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:36.798201 (+ 6781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:09:36.798208 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:09:36.798213 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:09:36.798254 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:09:36.801854 (+ 3600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:36.802653 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:36.802659 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:36.802662 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:36.802807 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:36.803253 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:36.803258 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:36.803262 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:36.803597 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:09:36.803866 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0926 15:09:36.804386 (+ 520us) server_negotiation.cc:299] Negotiation successful
0926 15:09:36.804597 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 15:09:36.979898 18572 catalog_manager.cc:4841] TS d3e156f17705476c90ae1f60578daa75 (127.12.34.193:38979): tablet 07023488e9454f0e9edc8faedd076ec8 (table test-table [id=745cd4d673724f5ab276c3dfffb91f76]) successfully deleted
I20260926 15:09:36.980278 18572 catalog_manager.cc:4841] TS 85dd0480684b43909497b35c05e43908 (127.12.34.194:38821): tablet 07023488e9454f0e9edc8faedd076ec8 (table test-table [id=745cd4d673724f5ab276c3dfffb91f76]) successfully deleted
I20260926 15:09:36.980505 12427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8uoJGv/build/asan/bin/kudu with pid 18556
I20260926 15:09:37.040474 19847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:37.015402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44562 (local address 127.12.34.253:32783)
0926 15:09:37.015611 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:37.015617 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:37.023957 (+ 8340us) server_negotiation.cc:407] Connection header received
0926 15:09:37.024720 (+ 763us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:37.024727 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:37.024810 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:37.024904 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:09:37.026511 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:37.027664 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:37.029395 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:37.029911 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:37.030046 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:37.033143 (+ 3097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:09:37.033149 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:09:37.033154 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:09:37.033193 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:09:37.035894 (+ 2701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:37.036498 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:37.036504 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:37.036507 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:37.036713 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:37.038794 (+ 2081us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:37.038801 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:37.038805 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:37.039160 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:09:37.039424 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0926 15:09:37.039944 (+ 520us) server_negotiation.cc:299] Negotiation successful
0926 15:09:37.040156 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 15:09:37.041052 19848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:37.023785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54126 (local address 127.12.34.252:44557)
0926 15:09:37.023986 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:37.023992 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:37.024468 (+ 476us) server_negotiation.cc:407] Connection header received
0926 15:09:37.024664 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:37.024672 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:37.024757 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:37.024865 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:09:37.027772 (+ 2907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:37.028871 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:37.030876 (+ 2005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:37.031581 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:37.031710 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:37.034816 (+ 3106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:09:37.034822 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:09:37.034827 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:09:37.034865 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:09:37.038263 (+ 3398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:37.039013 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:37.039018 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:37.039024 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:37.039156 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:37.039590 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:37.039595 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:37.039598 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:37.039930 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:09:37.040157 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0926 15:09:37.040685 (+ 528us) server_negotiation.cc:299] Negotiation successful
0926 15:09:37.040845 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 15:09:37.702034 19910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:37.701202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:42537 (local address 127.12.34.194:57549)
0926 15:09:37.701669 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0926 15:09:37.701807 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.254:42537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":261,"threads_started":1}
W20260926 15:09:37.702419 19703 heartbeater.cc:644] Failed to heartbeat to 127.12.34.254:42537 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.12.34.254:42537: connect: Connection refused (error 111)
I20260926 15:09:37.704412 19911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:37.703546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:42537 (local address 127.12.34.193:37235)
0926 15:09:37.704021 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0926 15:09:37.704192 (+ 171us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.254:42537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":248,"threads_started":1}
W20260926 15:09:37.704875 19555 heartbeater.cc:644] Failed to heartbeat to 127.12.34.254:42537 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.12.34.254:42537: connect: Connection refused (error 111)
I20260926 15:09:37.725292 19912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:37.724566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:42537 (local address 127.12.34.195:51441)
0926 15:09:37.724941 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0926 15:09:37.725066 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.254:42537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":196,"threads_started":1}
W20260926 15:09:37.725680 19844 heartbeater.cc:644] Failed to heartbeat to 127.12.34.254:42537 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.12.34.254:42537: connect: Connection refused (error 111)
I20260926 15:09:38.220422 19913 raft_consensus.cc:482] T 00000000000000000000000000000000 P 70d9c6c1008544b4b7ba623a53d9e680 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f8c077ce1f004166989faaf5668ef1e8)
I20260926 15:09:38.220734 19913 raft_consensus.cc:504] T 00000000000000000000000000000000 P 70d9c6c1008544b4b7ba623a53d9e680 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f8c077ce1f004166989faaf5668ef1e8" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 42537 } } peers { permanent_uuid: "f183e2953d3141608ce600f9167cd229" member_type: VOTER last_known_addr { host: "127.12.34.253" port: 32783 } attrs { promote: false } } peers { permanent_uuid: "70d9c6c1008544b4b7ba623a53d9e680" member_type: VOTER last_known_addr { host: "127.12.34.252" port: 44557 } attrs { promote: false } }
I20260926 15:09:38.220853 19914 raft_consensus.cc:482] T 00000000000000000000000000000000 P f183e2953d3141608ce600f9167cd229 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f8c077ce1f004166989faaf5668ef1e8)
I20260926 15:09:38.221181 19914 raft_consensus.cc:504] T 00000000000000000000000000000000 P f183e2953d3141608ce600f9167cd229 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f8c077ce1f004166989faaf5668ef1e8" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 42537 } } peers { permanent_uuid: "f183e2953d3141608ce600f9167cd229" member_type: VOTER last_known_addr { host: "127.12.34.253" port: 32783 } attrs { promote: false } } peers { permanent_uuid: "70d9c6c1008544b4b7ba623a53d9e680" member_type: VOTER last_known_addr { host: "127.12.34.252" port: 44557 } attrs { promote: false } }
I20260926 15:09:38.222112 19913 leader_election.cc:288] T 00000000000000000000000000000000 P 70d9c6c1008544b4b7ba623a53d9e680 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f8c077ce1f004166989faaf5668ef1e8 (127.12.34.254:42537), f183e2953d3141608ce600f9167cd229 (127.12.34.253:32783)
I20260926 15:09:38.228407 19914 leader_election.cc:288] T 00000000000000000000000000000000 P f183e2953d3141608ce600f9167cd229 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f8c077ce1f004166989faaf5668ef1e8 (127.12.34.254:42537), 70d9c6c1008544b4b7ba623a53d9e680 (127.12.34.252:44557)
I20260926 15:09:38.234510 19917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:38.222615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:42537 (local address 127.0.0.1:37264)
0926 15:09:38.234050 (+ 11435us) negotiation.cc:107] Waiting for socket to connect
0926 15:09:38.234268 (+ 218us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.254:42537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":11329,"thread_start_us":11196,"threads_started":1}
I20260926 15:09:38.234519 19915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:38.222196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:42537 (local address 127.0.0.1:37258)
0926 15:09:38.234051 (+ 11855us) negotiation.cc:107] Waiting for socket to connect
0926 15:09:38.234236 (+ 185us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.254:42537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":11749,"threads_started":1}
W20260926 15:09:38.234989 19361 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.34.254:42537: connect: Connection refused (error 111)
W20260926 15:09:38.234975 19173 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.34.254:42537: connect: Connection refused (error 111)
I20260926 15:09:38.245055 19915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:38.244615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:42537 (local address 127.0.0.1:37272)
0926 15:09:38.244795 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0926 15:09:38.244891 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.254:42537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260926 15:09:38.245303 19361 leader_election.cc:334] T 00000000000000000000000000000000 P 70d9c6c1008544b4b7ba623a53d9e680 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f8c077ce1f004166989faaf5668ef1e8 (127.12.34.254:42537): Network error: Client connection negotiation failed: client connection to 127.12.34.254:42537: connect: Connection refused (error 111)
I20260926 15:09:38.246498 19922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:38.245120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:42537 (local address 127.0.0.1:37282)
0926 15:09:38.246049 (+ 929us) negotiation.cc:107] Waiting for socket to connect
0926 15:09:38.246208 (+ 159us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.254:42537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":801,"thread_start_us":557,"threads_started":1}
W20260926 15:09:38.246718 19173 leader_election.cc:334] T 00000000000000000000000000000000 P f183e2953d3141608ce600f9167cd229 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f8c077ce1f004166989faaf5668ef1e8 (127.12.34.254:42537): Network error: Client connection negotiation failed: client connection to 127.12.34.254:42537: connect: Connection refused (error 111)
I20260926 15:09:38.246971 19918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:38.222616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44578 (local address 127.12.34.253:32783)
0926 15:09:38.233612 (+ 10996us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:38.233622 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:38.235329 (+ 1707us) server_negotiation.cc:407] Connection header received
0926 15:09:38.235556 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:38.235564 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:38.235650 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:38.235771 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:09:38.236821 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:38.238793 (+ 1972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:38.244086 (+ 5293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:38.245695 (+ 1609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:38.245866 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:38.245994 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0926 15:09:38.246465 (+ 471us) server_negotiation.cc:299] Negotiation successful
0926 15:09:38.246600 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10814,"thread_start_us":10989,"threads_started":1}
I20260926 15:09:38.247305 19916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:38.222280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.253:32783 (local address 127.0.0.1:44578)
0926 15:09:38.235222 (+ 12942us) negotiation.cc:107] Waiting for socket to connect
0926 15:09:38.235246 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 15:09:38.235412 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:09:38.235785 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:09:38.235792 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:09:38.235834 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:09:38.236451 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:38.236474 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:38.239293 (+ 2819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:38.239313 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:38.243196 (+ 3883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:38.243219 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:38.246051 (+ 2832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:38.246058 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:38.246270 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:38.246309 (+ 39us) client_negotiation.cc:769] Sending connection context
0926 15:09:38.246414 (+ 105us) client_negotiation.cc:240] Negotiation successful
0926 15:09:38.246678 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12771,"thread_start_us":13256,"threads_started":1}
I20260926 15:09:38.248090 19196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "70d9c6c1008544b4b7ba623a53d9e680" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "f183e2953d3141608ce600f9167cd229" is_pre_election: true
I20260926 15:09:38.248445 19196 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f183e2953d3141608ce600f9167cd229 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 70d9c6c1008544b4b7ba623a53d9e680 in term 1.
I20260926 15:09:38.248970 19362 leader_election.cc:302] T 00000000000000000000000000000000 P 70d9c6c1008544b4b7ba623a53d9e680 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 70d9c6c1008544b4b7ba623a53d9e680, f183e2953d3141608ce600f9167cd229; no voters: f8c077ce1f004166989faaf5668ef1e8
I20260926 15:09:38.249423 19913 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 70d9c6c1008544b4b7ba623a53d9e680 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260926 15:09:38.249591 19913 raft_consensus.cc:482] T 00000000000000000000000000000000 P 70d9c6c1008544b4b7ba623a53d9e680 [term 1 FOLLOWER]: Starting leader election (detected failure of leader f8c077ce1f004166989faaf5668ef1e8)
I20260926 15:09:38.249730 19913 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 70d9c6c1008544b4b7ba623a53d9e680 [term 1 FOLLOWER]: Advancing to term 2
I20260926 15:09:38.251157 19917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:38.234773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.252:44557 (local address 127.0.0.1:54138)
0926 15:09:38.235978 (+ 1205us) negotiation.cc:107] Waiting for socket to connect
0926 15:09:38.235998 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 15:09:38.236102 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:09:38.244954 (+ 8852us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:09:38.244961 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:09:38.244996 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:09:38.245380 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:38.245393 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:38.247079 (+ 1686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:38.247087 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:38.248983 (+ 1896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:38.248994 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:38.250478 (+ 1484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:38.250485 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:38.250650 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:38.250673 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 15:09:38.250777 (+ 104us) client_negotiation.cc:240] Negotiation successful
0926 15:09:38.250907 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":584}
I20260926 15:09:38.251358 19919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:38.234525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54138 (local address 127.12.34.252:44557)
0926 15:09:38.243822 (+ 9297us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:38.243833 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:38.243852 (+ 19us) server_negotiation.cc:407] Connection header received
0926 15:09:38.243916 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:38.243920 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:38.244015 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:38.245525 (+ 1510us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:09:38.245770 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:38.246844 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:38.249163 (+ 2319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:38.250295 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:38.250441 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:38.250548 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 15:09:38.250809 (+ 261us) server_negotiation.cc:299] Negotiation successful
0926 15:09:38.250935 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9143,"thread_start_us":8989,"threads_started":1}
I20260926 15:09:38.251966 19385 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f183e2953d3141608ce600f9167cd229" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "70d9c6c1008544b4b7ba623a53d9e680" is_pre_election: true
I20260926 15:09:38.252370 19913 raft_consensus.cc:504] T 00000000000000000000000000000000 P 70d9c6c1008544b4b7ba623a53d9e680 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f8c077ce1f004166989faaf5668ef1e8" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 42537 } } peers { permanent_uuid: "f183e2953d3141608ce600f9167cd229" member_type: VOTER last_known_addr { host: "127.12.34.253" port: 32783 } attrs { promote: false } } peers { permanent_uuid: "70d9c6c1008544b4b7ba623a53d9e680" member_type: VOTER last_known_addr { host: "127.12.34.252" port: 44557 } attrs { promote: false } }
I20260926 15:09:38.252774 19385 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 70d9c6c1008544b4b7ba623a53d9e680 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f183e2953d3141608ce600f9167cd229 in current term 2: Already voted for candidate 70d9c6c1008544b4b7ba623a53d9e680 in this term.
I20260926 15:09:38.253010 19913 leader_election.cc:288] T 00000000000000000000000000000000 P 70d9c6c1008544b4b7ba623a53d9e680 [CANDIDATE]: Term 2 election: Requested vote from peers f8c077ce1f004166989faaf5668ef1e8 (127.12.34.254:42537), f183e2953d3141608ce600f9167cd229 (127.12.34.253:32783)
I20260926 15:09:38.253499 19916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:38.253124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:42537 (local address 127.0.0.1:37288)
0926 15:09:38.253271 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0926 15:09:38.253352 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.254:42537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260926 15:09:38.253453 19196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "70d9c6c1008544b4b7ba623a53d9e680" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "f183e2953d3141608ce600f9167cd229"
I20260926 15:09:38.253639 19196 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f183e2953d3141608ce600f9167cd229 [term 1 FOLLOWER]: Advancing to term 2
I20260926 15:09:38.253698 19173 leader_election.cc:302] T 00000000000000000000000000000000 P f183e2953d3141608ce600f9167cd229 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f183e2953d3141608ce600f9167cd229; no voters: 70d9c6c1008544b4b7ba623a53d9e680, f8c077ce1f004166989faaf5668ef1e8
I20260926 15:09:38.254468 19916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:38.254109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:42537 (local address 127.0.0.1:37300)
0926 15:09:38.254285 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0926 15:09:38.254360 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.254:42537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
W20260926 15:09:38.254663 19361 leader_election.cc:334] T 00000000000000000000000000000000 P 70d9c6c1008544b4b7ba623a53d9e680 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer f8c077ce1f004166989faaf5668ef1e8 (127.12.34.254:42537): Network error: Client connection negotiation failed: client connection to 127.12.34.254:42537: connect: Connection refused (error 111)
I20260926 15:09:38.255797 19196 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f183e2953d3141608ce600f9167cd229 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 70d9c6c1008544b4b7ba623a53d9e680 in term 2.
I20260926 15:09:38.255954 19914 raft_consensus.cc:2726] T 00000000000000000000000000000000 P f183e2953d3141608ce600f9167cd229 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 15:09:38.256282 19362 leader_election.cc:302] T 00000000000000000000000000000000 P 70d9c6c1008544b4b7ba623a53d9e680 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 70d9c6c1008544b4b7ba623a53d9e680, f183e2953d3141608ce600f9167cd229; no voters: f8c077ce1f004166989faaf5668ef1e8
I20260926 15:09:38.256567 19913 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 70d9c6c1008544b4b7ba623a53d9e680 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 15:09:38.256975 19913 raft_consensus.cc:686] T 00000000000000000000000000000000 P 70d9c6c1008544b4b7ba623a53d9e680 [term 2 LEADER]: Becoming Leader. State: Replica: 70d9c6c1008544b4b7ba623a53d9e680, State: Running, Role: LEADER
I20260926 15:09:38.257341 19913 consensus_queue.cc:237] T 00000000000000000000000000000000 P 70d9c6c1008544b4b7ba623a53d9e680 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f8c077ce1f004166989faaf5668ef1e8" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 42537 } } peers { permanent_uuid: "f183e2953d3141608ce600f9167cd229" member_type: VOTER last_known_addr { host: "127.12.34.253" port: 32783 } attrs { promote: false } } peers { permanent_uuid: "70d9c6c1008544b4b7ba623a53d9e680" member_type: VOTER last_known_addr { host: "127.12.34.252" port: 44557 } attrs { promote: false } }
I20260926 15:09:38.259244 19923 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70d9c6c1008544b4b7ba623a53d9e680 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 70d9c6c1008544b4b7ba623a53d9e680. Latest consensus state: current_term: 2 leader_uuid: "70d9c6c1008544b4b7ba623a53d9e680" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f8c077ce1f004166989faaf5668ef1e8" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 42537 } } peers { permanent_uuid: "f183e2953d3141608ce600f9167cd229" member_type: VOTER last_known_addr { host: "127.12.34.253" port: 32783 } attrs { promote: false } } peers { permanent_uuid: "70d9c6c1008544b4b7ba623a53d9e680" member_type: VOTER last_known_addr { host: "127.12.34.252" port: 44557 } attrs { promote: false } } }
I20260926 15:09:38.259692 19923 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70d9c6c1008544b4b7ba623a53d9e680 [sys.catalog]: This master's current role is: LEADER
I20260926 15:09:38.260344 19925 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:09:38.263975 19925 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 15:09:38.264870 19925 catalog_manager.cc:1259] Loaded cluster ID: 165375e664a14f989e39db1da9eac0b9
I20260926 15:09:38.264940 19925 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 15:09:38.266183 19925 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:09:38.267149 19925 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 70d9c6c1008544b4b7ba623a53d9e680: Loaded TSK: 0
I20260926 15:09:38.268244 19925 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 15:09:38.320495 19374 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54126:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 15:09:38.327382 19374 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 15:09:38.331674 19916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:38.331302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:42537 (local address 127.0.0.1:37334)
0926 15:09:38.331477 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0926 15:09:38.331547 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.254:42537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260926 15:09:38.331704 19196 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f183e2953d3141608ce600f9167cd229 [term 2 FOLLOWER]: Refusing update from remote peer 70d9c6c1008544b4b7ba623a53d9e680: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260926 15:09:38.332541 19923 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 70d9c6c1008544b4b7ba623a53d9e680 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f183e2953d3141608ce600f9167cd229" member_type: VOTER last_known_addr { host: "127.12.34.253" port: 32783 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260926 15:09:38.332746 19916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:38.332206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:42537 (local address 127.0.0.1:37340)
0926 15:09:38.332445 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0926 15:09:38.332543 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.254:42537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260926 15:09:38.332933 19361 consensus_peers.cc:487] T 00000000000000000000000000000000 P 70d9c6c1008544b4b7ba623a53d9e680 -> Peer f8c077ce1f004166989faaf5668ef1e8 (127.12.34.254:42537): Couldn't send request to peer f8c077ce1f004166989faaf5668ef1e8. Status: Network error: Client connection negotiation failed: client connection to 127.12.34.254:42537: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260926 15:09:38.335158 19914 sys_catalog.cc:455] T 00000000000000000000000000000000 P f183e2953d3141608ce600f9167cd229 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 70d9c6c1008544b4b7ba623a53d9e680. Latest consensus state: current_term: 2 leader_uuid: "70d9c6c1008544b4b7ba623a53d9e680" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f8c077ce1f004166989faaf5668ef1e8" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 42537 } } peers { permanent_uuid: "f183e2953d3141608ce600f9167cd229" member_type: VOTER last_known_addr { host: "127.12.34.253" port: 32783 } attrs { promote: false } } peers { permanent_uuid: "70d9c6c1008544b4b7ba623a53d9e680" member_type: VOTER last_known_addr { host: "127.12.34.252" port: 44557 } attrs { promote: false } } }
I20260926 15:09:38.335508 19914 sys_catalog.cc:458] T 00000000000000000000000000000000 P f183e2953d3141608ce600f9167cd229 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 15:09:38.336880 19913 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70d9c6c1008544b4b7ba623a53d9e680 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "70d9c6c1008544b4b7ba623a53d9e680" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f8c077ce1f004166989faaf5668ef1e8" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 42537 } } peers { permanent_uuid: "f183e2953d3141608ce600f9167cd229" member_type: VOTER last_known_addr { host: "127.12.34.253" port: 32783 } attrs { promote: false } } peers { permanent_uuid: "70d9c6c1008544b4b7ba623a53d9e680" member_type: VOTER last_known_addr { host: "127.12.34.252" port: 44557 } attrs { promote: false } } }
I20260926 15:09:38.337272 19913 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70d9c6c1008544b4b7ba623a53d9e680 [sys.catalog]: This master's current role is: LEADER
I20260926 15:09:38.337709 19914 sys_catalog.cc:455] T 00000000000000000000000000000000 P f183e2953d3141608ce600f9167cd229 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "70d9c6c1008544b4b7ba623a53d9e680" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f8c077ce1f004166989faaf5668ef1e8" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 42537 } } peers { permanent_uuid: "f183e2953d3141608ce600f9167cd229" member_type: VOTER last_known_addr { host: "127.12.34.253" port: 32783 } attrs { promote: false } } peers { permanent_uuid: "70d9c6c1008544b4b7ba623a53d9e680" member_type: VOTER last_known_addr { host: "127.12.34.252" port: 44557 } attrs { promote: false } } }
I20260926 15:09:38.337949 19914 sys_catalog.cc:458] T 00000000000000000000000000000000 P f183e2953d3141608ce600f9167cd229 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 15:09:38.362433 19916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:38.344923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.194:38821 (local address 127.0.0.1:40480)
0926 15:09:38.345149 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0926 15:09:38.345168 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 15:09:38.345280 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:09:38.357376 (+ 12096us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:09:38.357380 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:09:38.357403 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:09:38.357598 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:38.357609 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:38.358607 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:38.358614 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:38.360150 (+ 1536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:38.360162 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:38.361743 (+ 1581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:38.361748 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:38.361925 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:38.361960 (+ 35us) client_negotiation.cc:769] Sending connection context
0926 15:09:38.362022 (+ 62us) client_negotiation.cc:240] Negotiation successful
0926 15:09:38.362152 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89,"mutex_wait_us":90}
I20260926 15:09:38.363462 19915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:38.345277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.195:36967 (local address 127.0.0.1:38926)
0926 15:09:38.345474 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0926 15:09:38.345490 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 15:09:38.345593 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:09:38.356716 (+ 11123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:09:38.356722 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:09:38.356767 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:09:38.357056 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:38.357065 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:38.358524 (+ 1459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:38.358531 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:38.360684 (+ 2153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:38.360697 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:38.362736 (+ 2039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:38.362742 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:38.362918 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:38.362966 (+ 48us) client_negotiation.cc:769] Sending connection context
0926 15:09:38.363035 (+ 69us) client_negotiation.cc:240] Negotiation successful
0926 15:09:38.363175 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"mutex_wait_us":3}
I20260926 15:09:38.363718 19932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:38.344855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40480 (local address 127.12.34.194:38821)
0926 15:09:38.357046 (+ 12191us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:38.357051 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:38.357066 (+ 15us) server_negotiation.cc:407] Connection header received
0926 15:09:38.357175 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:38.357181 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:38.357250 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:38.357361 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:09:38.357754 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:38.358409 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:38.360387 (+ 1978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:38.361568 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:38.362941 (+ 1373us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:38.363035 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 15:09:38.363082 (+ 47us) server_negotiation.cc:299] Negotiation successful
0926 15:09:38.363256 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12086,"thread_start_us":12592,"threads_started":1}
I20260926 15:09:38.364099 19934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:38.345521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.193:38979 (local address 127.0.0.1:57018)
0926 15:09:38.346072 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0926 15:09:38.346090 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 15:09:38.346185 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:09:38.356439 (+ 10254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:09:38.356444 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:09:38.356472 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:09:38.356897 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:38.356914 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:38.358526 (+ 1612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:38.358537 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:38.360906 (+ 2369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:38.360919 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:38.363066 (+ 2147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:38.363072 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:38.363219 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:38.363248 (+ 29us) client_negotiation.cc:769] Sending connection context
0926 15:09:38.363307 (+ 59us) client_negotiation.cc:240] Negotiation successful
0926 15:09:38.363415 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"mutex_wait_us":54,"thread_start_us":526,"threads_started":1}
I20260926 15:09:38.364292 19933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:38.345357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38926 (local address 127.12.34.195:36967)
0926 15:09:38.356383 (+ 11026us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:38.356391 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:38.356410 (+ 19us) server_negotiation.cc:407] Connection header received
0926 15:09:38.356483 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:38.356487 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:38.356564 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:38.357202 (+ 638us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:09:38.357380 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:38.358378 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:38.361320 (+ 2942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:38.362528 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:38.363679 (+ 1151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:38.363784 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 15:09:38.363833 (+ 49us) server_negotiation.cc:299] Negotiation successful
0926 15:09:38.363959 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10896,"thread_start_us":10790,"threads_started":1}
I20260926 15:09:38.364941 19935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:38.345984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57018 (local address 127.12.34.193:38979)
0926 15:09:38.356002 (+ 10018us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:38.356008 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:38.356028 (+ 20us) server_negotiation.cc:407] Connection header received
0926 15:09:38.356104 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:38.356112 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:38.356203 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:38.357066 (+ 863us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:09:38.357260 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:38.358371 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:38.361097 (+ 2726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:38.362849 (+ 1752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:38.364354 (+ 1505us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:38.364464 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0926 15:09:38.364514 (+ 50us) server_negotiation.cc:299] Negotiation successful
0926 15:09:38.364648 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9885,"thread_start_us":10303,"threads_started":1}
I20260926 15:09:38.364974 19637 tablet_service.cc:1467] Processing CreateTablet for tablet a273bedf1df042cdb7546e04fcbdaeff (DEFAULT_TABLE table=test-table [id=8c3b946f89304340b815c70025d3f900]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:09:38.364990 19778 tablet_service.cc:1467] Processing CreateTablet for tablet a273bedf1df042cdb7546e04fcbdaeff (DEFAULT_TABLE table=test-table [id=8c3b946f89304340b815c70025d3f900]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:09:38.365423 19778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a273bedf1df042cdb7546e04fcbdaeff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:09:38.365442 19637 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a273bedf1df042cdb7546e04fcbdaeff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:09:38.369732 19489 tablet_service.cc:1467] Processing CreateTablet for tablet a273bedf1df042cdb7546e04fcbdaeff (DEFAULT_TABLE table=test-table [id=8c3b946f89304340b815c70025d3f900]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:09:38.370179 19489 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a273bedf1df042cdb7546e04fcbdaeff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:09:38.372090 19936 tablet_bootstrap.cc:492] T a273bedf1df042cdb7546e04fcbdaeff P 71e96bd442d34587b6d0e98bb1c8aae4: Bootstrap starting.
I20260926 15:09:38.373813 19936 tablet_bootstrap.cc:654] T a273bedf1df042cdb7546e04fcbdaeff P 71e96bd442d34587b6d0e98bb1c8aae4: Neither blocks nor log segments found. Creating new log.
I20260926 15:09:38.374763 19937 tablet_bootstrap.cc:492] T a273bedf1df042cdb7546e04fcbdaeff P 85dd0480684b43909497b35c05e43908: Bootstrap starting.
I20260926 15:09:38.376405 19938 tablet_bootstrap.cc:492] T a273bedf1df042cdb7546e04fcbdaeff P d3e156f17705476c90ae1f60578daa75: Bootstrap starting.
I20260926 15:09:38.376432 19937 tablet_bootstrap.cc:654] T a273bedf1df042cdb7546e04fcbdaeff P 85dd0480684b43909497b35c05e43908: Neither blocks nor log segments found. Creating new log.
I20260926 15:09:38.377686 19938 tablet_bootstrap.cc:654] T a273bedf1df042cdb7546e04fcbdaeff P d3e156f17705476c90ae1f60578daa75: Neither blocks nor log segments found. Creating new log.
I20260926 15:09:38.379061 19937 tablet_bootstrap.cc:492] T a273bedf1df042cdb7546e04fcbdaeff P 85dd0480684b43909497b35c05e43908: No bootstrap required, opened a new log
I20260926 15:09:38.379218 19937 ts_tablet_manager.cc:1397] T a273bedf1df042cdb7546e04fcbdaeff P 85dd0480684b43909497b35c05e43908: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260926 15:09:38.379591 19938 tablet_bootstrap.cc:492] T a273bedf1df042cdb7546e04fcbdaeff P d3e156f17705476c90ae1f60578daa75: No bootstrap required, opened a new log
I20260926 15:09:38.379737 19938 ts_tablet_manager.cc:1397] T a273bedf1df042cdb7546e04fcbdaeff P d3e156f17705476c90ae1f60578daa75: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260926 15:09:38.379941 19937 raft_consensus.cc:348] T a273bedf1df042cdb7546e04fcbdaeff P 85dd0480684b43909497b35c05e43908 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85dd0480684b43909497b35c05e43908" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 38821 } } peers { permanent_uuid: "71e96bd442d34587b6d0e98bb1c8aae4" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 36967 } } peers { permanent_uuid: "d3e156f17705476c90ae1f60578daa75" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 38979 } }
I20260926 15:09:38.380111 19937 raft_consensus.cc:374] T a273bedf1df042cdb7546e04fcbdaeff P 85dd0480684b43909497b35c05e43908 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:09:38.380213 19937 raft_consensus.cc:729] T a273bedf1df042cdb7546e04fcbdaeff P 85dd0480684b43909497b35c05e43908 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85dd0480684b43909497b35c05e43908, State: Initialized, Role: FOLLOWER
I20260926 15:09:38.380438 19937 consensus_queue.cc:260] T a273bedf1df042cdb7546e04fcbdaeff P 85dd0480684b43909497b35c05e43908 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85dd0480684b43909497b35c05e43908" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 38821 } } peers { permanent_uuid: "71e96bd442d34587b6d0e98bb1c8aae4" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 36967 } } peers { permanent_uuid: "d3e156f17705476c90ae1f60578daa75" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 38979 } }
I20260926 15:09:38.380467 19938 raft_consensus.cc:348] T a273bedf1df042cdb7546e04fcbdaeff P d3e156f17705476c90ae1f60578daa75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85dd0480684b43909497b35c05e43908" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 38821 } } peers { permanent_uuid: "71e96bd442d34587b6d0e98bb1c8aae4" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 36967 } } peers { permanent_uuid: "d3e156f17705476c90ae1f60578daa75" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 38979 } }
I20260926 15:09:38.380678 19938 raft_consensus.cc:374] T a273bedf1df042cdb7546e04fcbdaeff P d3e156f17705476c90ae1f60578daa75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:09:38.380785 19938 raft_consensus.cc:729] T a273bedf1df042cdb7546e04fcbdaeff P d3e156f17705476c90ae1f60578daa75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3e156f17705476c90ae1f60578daa75, State: Initialized, Role: FOLLOWER
I20260926 15:09:38.381282 19938 consensus_queue.cc:260] T a273bedf1df042cdb7546e04fcbdaeff P d3e156f17705476c90ae1f60578daa75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85dd0480684b43909497b35c05e43908" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 38821 } } peers { permanent_uuid: "71e96bd442d34587b6d0e98bb1c8aae4" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 36967 } } peers { permanent_uuid: "d3e156f17705476c90ae1f60578daa75" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 38979 } }
I20260926 15:09:38.381474 19936 tablet_bootstrap.cc:492] T a273bedf1df042cdb7546e04fcbdaeff P 71e96bd442d34587b6d0e98bb1c8aae4: No bootstrap required, opened a new log
I20260926 15:09:38.381604 19936 ts_tablet_manager.cc:1397] T a273bedf1df042cdb7546e04fcbdaeff P 71e96bd442d34587b6d0e98bb1c8aae4: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.003s
I20260926 15:09:38.382370 19936 raft_consensus.cc:348] T a273bedf1df042cdb7546e04fcbdaeff P 71e96bd442d34587b6d0e98bb1c8aae4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85dd0480684b43909497b35c05e43908" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 38821 } } peers { permanent_uuid: "71e96bd442d34587b6d0e98bb1c8aae4" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 36967 } } peers { permanent_uuid: "d3e156f17705476c90ae1f60578daa75" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 38979 } }
I20260926 15:09:38.382570 19936 raft_consensus.cc:374] T a273bedf1df042cdb7546e04fcbdaeff P 71e96bd442d34587b6d0e98bb1c8aae4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:09:38.382654 19936 raft_consensus.cc:729] T a273bedf1df042cdb7546e04fcbdaeff P 71e96bd442d34587b6d0e98bb1c8aae4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71e96bd442d34587b6d0e98bb1c8aae4, State: Initialized, Role: FOLLOWER
I20260926 15:09:38.383067 19936 consensus_queue.cc:260] T a273bedf1df042cdb7546e04fcbdaeff P 71e96bd442d34587b6d0e98bb1c8aae4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85dd0480684b43909497b35c05e43908" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 38821 } } peers { permanent_uuid: "71e96bd442d34587b6d0e98bb1c8aae4" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 36967 } } peers { permanent_uuid: "d3e156f17705476c90ae1f60578daa75" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 38979 } }
I20260926 15:09:38.383785 19937 ts_tablet_manager.cc:1428] T a273bedf1df042cdb7546e04fcbdaeff P 85dd0480684b43909497b35c05e43908: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20260926 15:09:38.392354 19936 ts_tablet_manager.cc:1428] T a273bedf1df042cdb7546e04fcbdaeff P 71e96bd442d34587b6d0e98bb1c8aae4: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20260926 15:09:38.393168 19843 heartbeater.cc:503] Master 127.12.34.252:44557 was elected leader, sending a full tablet report...
I20260926 15:09:38.393831 19946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:38.384558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:42537 (local address 127.12.34.195:50069)
0926 15:09:38.393010 (+ 8452us) negotiation.cc:107] Waiting for socket to connect
0926 15:09:38.393168 (+ 158us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.254:42537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":8310,"thread_start_us":8179,"threads_started":1}
I20260926 15:09:38.394377 19938 ts_tablet_manager.cc:1428] T a273bedf1df042cdb7546e04fcbdaeff P d3e156f17705476c90ae1f60578daa75: Time spent starting tablet: real 0.015s user 0.004s sys 0.007s
I20260926 15:09:38.395256 19944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:38.381524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:42537 (local address 127.12.34.194:48321)
0926 15:09:38.393690 (+ 12166us) negotiation.cc:107] Waiting for socket to connect
0926 15:09:38.393835 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.254:42537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":12030,"thread_start_us":11920,"threads_started":1}
I20260926 15:09:38.396347 19702 heartbeater.cc:503] Master 127.12.34.252:44557 was elected leader, sending a full tablet report...
I20260926 15:09:38.396808 19946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:38.395935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:42537 (local address 127.12.34.195:35479)
0926 15:09:38.396574 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0926 15:09:38.396666 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.254:42537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":540}
I20260926 15:09:38.397903 19554 heartbeater.cc:503] Master 127.12.34.252:44557 was elected leader, sending a full tablet report...
I20260926 15:09:38.398005 19944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:38.397061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:42537 (local address 127.12.34.194:37889)
0926 15:09:38.397741 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0926 15:09:38.397823 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.254:42537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":579}
I20260926 15:09:38.398366 19947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:38.395014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:42537 (local address 127.12.34.193:37359)
0926 15:09:38.397746 (+ 2732us) negotiation.cc:107] Waiting for socket to connect
0926 15:09:38.397901 (+ 155us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.254:42537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2591,"thread_start_us":2463,"threads_started":1}
I20260926 15:09:38.399283 19947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:38.398982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:42537 (local address 127.12.34.193:56411)
0926 15:09:38.399114 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0926 15:09:38.399193 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.254:42537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260926 15:09:38.514086 19705 tablet.cc:2369] T a273bedf1df042cdb7546e04fcbdaeff P 85dd0480684b43909497b35c05e43908: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 15:09:38.551709 19943 raft_consensus.cc:482] T a273bedf1df042cdb7546e04fcbdaeff P d3e156f17705476c90ae1f60578daa75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 15:09:38.551980 19943 raft_consensus.cc:504] T a273bedf1df042cdb7546e04fcbdaeff P d3e156f17705476c90ae1f60578daa75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85dd0480684b43909497b35c05e43908" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 38821 } } peers { permanent_uuid: "71e96bd442d34587b6d0e98bb1c8aae4" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 36967 } } peers { permanent_uuid: "d3e156f17705476c90ae1f60578daa75" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 38979 } }
I20260926 15:09:38.553544 19943 leader_election.cc:288] T a273bedf1df042cdb7546e04fcbdaeff P d3e156f17705476c90ae1f60578daa75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85dd0480684b43909497b35c05e43908 (127.12.34.194:38821), 71e96bd442d34587b6d0e98bb1c8aae4 (127.12.34.195:36967)
I20260926 15:09:38.561288 19933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:38.553960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.193:57519 (local address 127.12.34.195:36967)
0926 15:09:38.554180 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:38.554185 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:38.554511 (+ 326us) server_negotiation.cc:407] Connection header received
0926 15:09:38.554809 (+ 298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:38.554817 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:38.554902 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:38.555008 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:09:38.555626 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:38.556508 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:38.559286 (+ 2778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:38.560415 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:38.560576 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:38.560656 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 15:09:38.560912 (+ 256us) server_negotiation.cc:299] Negotiation successful
0926 15:09:38.561039 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 15:09:38.561367 19947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:38.553956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.195:36967 (local address 127.12.34.193:57519)
0926 15:09:38.554234 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0926 15:09:38.554398 (+ 164us) client_negotiation.cc:174] Beginning negotiation
0926 15:09:38.554589 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:09:38.555051 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:09:38.555055 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:09:38.555080 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:09:38.555465 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:38.555478 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:38.556638 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:38.556644 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:38.559131 (+ 2487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:38.559146 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:38.560604 (+ 1458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:38.560609 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:38.560750 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:38.560773 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 15:09:38.560855 (+ 82us) client_negotiation.cc:240] Negotiation successful
0926 15:09:38.561039 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"mutex_wait_us":24}
I20260926 15:09:38.561841 19932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:38.553946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.193:51245 (local address 127.12.34.194:38821)
0926 15:09:38.554176 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:38.554183 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:38.555161 (+ 978us) server_negotiation.cc:407] Connection header received
0926 15:09:38.555362 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:38.555370 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:38.555457 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:38.555571 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:09:38.556102 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:38.557032 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:38.559611 (+ 2579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:38.560811 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:38.560987 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:38.561050 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 15:09:38.561488 (+ 438us) server_negotiation.cc:299] Negotiation successful
0926 15:09:38.561640 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 15:09:38.562036 19798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a273bedf1df042cdb7546e04fcbdaeff" candidate_uuid: "d3e156f17705476c90ae1f60578daa75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71e96bd442d34587b6d0e98bb1c8aae4" is_pre_election: true
I20260926 15:09:38.562292 19798 raft_consensus.cc:2454] T a273bedf1df042cdb7546e04fcbdaeff P 71e96bd442d34587b6d0e98bb1c8aae4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3e156f17705476c90ae1f60578daa75 in term 0.
I20260926 15:09:38.562525 19948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:38.554238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.194:38821 (local address 127.12.34.193:51245)
0926 15:09:38.555049 (+ 811us) negotiation.cc:107] Waiting for socket to connect
0926 15:09:38.555067 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 15:09:38.555194 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:09:38.555602 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:09:38.555606 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:09:38.555628 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:09:38.555955 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:38.555965 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:38.557253 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:38.557259 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:38.559453 (+ 2194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:38.559463 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:38.561068 (+ 1605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:38.561095 (+ 27us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:38.561282 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:38.561313 (+ 31us) client_negotiation.cc:769] Sending connection context
0926 15:09:38.562037 (+ 724us) client_negotiation.cc:240] Negotiation successful
0926 15:09:38.562146 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":666,"mutex_wait_us":34,"thread_start_us":528,"threads_started":1}
I20260926 15:09:38.562832 19441 leader_election.cc:302] T a273bedf1df042cdb7546e04fcbdaeff P d3e156f17705476c90ae1f60578daa75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71e96bd442d34587b6d0e98bb1c8aae4, d3e156f17705476c90ae1f60578daa75; no voters:
I20260926 15:09:38.563299 19943 raft_consensus.cc:2781] T a273bedf1df042cdb7546e04fcbdaeff P d3e156f17705476c90ae1f60578daa75 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 15:09:38.563436 19943 raft_consensus.cc:482] T a273bedf1df042cdb7546e04fcbdaeff P d3e156f17705476c90ae1f60578daa75 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 15:09:38.563524 19943 raft_consensus.cc:3037] T a273bedf1df042cdb7546e04fcbdaeff P d3e156f17705476c90ae1f60578daa75 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:09:38.563411 19657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a273bedf1df042cdb7546e04fcbdaeff" candidate_uuid: "d3e156f17705476c90ae1f60578daa75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85dd0480684b43909497b35c05e43908" is_pre_election: true
I20260926 15:09:38.563835 19657 raft_consensus.cc:2454] T a273bedf1df042cdb7546e04fcbdaeff P 85dd0480684b43909497b35c05e43908 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3e156f17705476c90ae1f60578daa75 in term 0.
I20260926 15:09:38.566092 19943 raft_consensus.cc:504] T a273bedf1df042cdb7546e04fcbdaeff P d3e156f17705476c90ae1f60578daa75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85dd0480684b43909497b35c05e43908" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 38821 } } peers { permanent_uuid: "71e96bd442d34587b6d0e98bb1c8aae4" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 36967 } } peers { permanent_uuid: "d3e156f17705476c90ae1f60578daa75" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 38979 } }
I20260926 15:09:38.566838 19943 leader_election.cc:288] T a273bedf1df042cdb7546e04fcbdaeff P d3e156f17705476c90ae1f60578daa75 [CANDIDATE]: Term 1 election: Requested vote from peers 85dd0480684b43909497b35c05e43908 (127.12.34.194:38821), 71e96bd442d34587b6d0e98bb1c8aae4 (127.12.34.195:36967)
I20260926 15:09:38.567279 19657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a273bedf1df042cdb7546e04fcbdaeff" candidate_uuid: "d3e156f17705476c90ae1f60578daa75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85dd0480684b43909497b35c05e43908"
I20260926 15:09:38.567507 19657 raft_consensus.cc:3037] T a273bedf1df042cdb7546e04fcbdaeff P 85dd0480684b43909497b35c05e43908 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:09:38.569911 19657 raft_consensus.cc:2454] T a273bedf1df042cdb7546e04fcbdaeff P 85dd0480684b43909497b35c05e43908 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3e156f17705476c90ae1f60578daa75 in term 1.
I20260926 15:09:38.570322 19440 leader_election.cc:302] T a273bedf1df042cdb7546e04fcbdaeff P d3e156f17705476c90ae1f60578daa75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85dd0480684b43909497b35c05e43908, d3e156f17705476c90ae1f60578daa75; no voters:
I20260926 15:09:38.570636 19943 raft_consensus.cc:2781] T a273bedf1df042cdb7546e04fcbdaeff P d3e156f17705476c90ae1f60578daa75 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:09:38.571184 19943 raft_consensus.cc:686] T a273bedf1df042cdb7546e04fcbdaeff P d3e156f17705476c90ae1f60578daa75 [term 1 LEADER]: Becoming Leader. State: Replica: d3e156f17705476c90ae1f60578daa75, State: Running, Role: LEADER
I20260926 15:09:38.571592 19943 consensus_queue.cc:237] T a273bedf1df042cdb7546e04fcbdaeff P d3e156f17705476c90ae1f60578daa75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85dd0480684b43909497b35c05e43908" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 38821 } } peers { permanent_uuid: "71e96bd442d34587b6d0e98bb1c8aae4" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 36967 } } peers { permanent_uuid: "d3e156f17705476c90ae1f60578daa75" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 38979 } }
I20260926 15:09:38.572662 19948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:38.572179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:42537 (local address 127.12.34.193:60487)
0926 15:09:38.572386 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0926 15:09:38.572502 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.254:42537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
I20260926 15:09:38.572374 19798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a273bedf1df042cdb7546e04fcbdaeff" candidate_uuid: "d3e156f17705476c90ae1f60578daa75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71e96bd442d34587b6d0e98bb1c8aae4"
I20260926 15:09:38.573148 19798 raft_consensus.cc:3037] T a273bedf1df042cdb7546e04fcbdaeff P 71e96bd442d34587b6d0e98bb1c8aae4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:09:38.575696 19374 catalog_manager.cc:5495] T a273bedf1df042cdb7546e04fcbdaeff P d3e156f17705476c90ae1f60578daa75 reported cstate change: term changed from 0 to 1, leader changed from <none> to d3e156f17705476c90ae1f60578daa75 (127.12.34.193). New cstate: current_term: 1 leader_uuid: "d3e156f17705476c90ae1f60578daa75" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85dd0480684b43909497b35c05e43908" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 38821 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "71e96bd442d34587b6d0e98bb1c8aae4" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 36967 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3e156f17705476c90ae1f60578daa75" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 38979 } health_report { overall_health: HEALTHY } } }
I20260926 15:09:38.577572 19798 raft_consensus.cc:2454] T a273bedf1df042cdb7546e04fcbdaeff P 71e96bd442d34587b6d0e98bb1c8aae4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3e156f17705476c90ae1f60578daa75 in term 1.
I20260926 15:09:38.647925 19935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:38.642545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57032 (local address 127.12.34.193:38979)
0926 15:09:38.642833 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:38.642839 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:38.642865 (+ 26us) server_negotiation.cc:407] Connection header received
0926 15:09:38.643118 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:38.643124 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:38.643217 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:38.643342 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 15:09:38.643638 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:38.644320 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:38.645678 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:38.646084 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:38.646177 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:38.646613 (+ 436us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 15:09:38.647299 (+ 686us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 15:09:38.647481 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0926 15:09:38.647529 (+ 48us) server_negotiation.cc:299] Negotiation successful
0926 15:09:38.647595 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111}
I20260926 15:09:38.652161 19798 raft_consensus.cc:1261] T a273bedf1df042cdb7546e04fcbdaeff P 71e96bd442d34587b6d0e98bb1c8aae4 [term 1 FOLLOWER]: Refusing update from remote peer d3e156f17705476c90ae1f60578daa75: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:09:38.652263 19657 raft_consensus.cc:1261] T a273bedf1df042cdb7546e04fcbdaeff P 85dd0480684b43909497b35c05e43908 [term 1 FOLLOWER]: Refusing update from remote peer d3e156f17705476c90ae1f60578daa75: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:09:38.652874 19943 consensus_queue.cc:1035] T a273bedf1df042cdb7546e04fcbdaeff P d3e156f17705476c90ae1f60578daa75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71e96bd442d34587b6d0e98bb1c8aae4" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 36967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:09:38.653177 19949 consensus_queue.cc:1035] T a273bedf1df042cdb7546e04fcbdaeff P d3e156f17705476c90ae1f60578daa75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85dd0480684b43909497b35c05e43908" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 38821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:09:38.655503 19944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:38.655043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:42537 (local address 127.12.34.194:44373)
0926 15:09:38.655196 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0926 15:09:38.655308 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.254:42537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260926 15:09:38.655711 19946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:38.655207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:42537 (local address 127.12.34.195:45513)
0926 15:09:38.655386 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0926 15:09:38.655516 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.254:42537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260926 15:09:38.661607 19944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:38.661067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:42537 (local address 127.12.34.194:46087)
0926 15:09:38.661328 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0926 15:09:38.661410 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.254:42537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137}
I20260926 15:09:38.661586 19952 mvcc.cc:205] Tried to move back new op lower bound from 7333623310951587840 to 7333623310633447424. Current Snapshot: MvccSnapshot[applied={T|T < 7333623310951587840}]
I20260926 15:09:38.662873 19946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:38.662304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:42537 (local address 127.12.34.195:48583)
0926 15:09:38.662558 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0926 15:09:38.662661 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.254:42537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126}
I20260926 15:09:38.667654 19948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:38.667124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:42537 (local address 127.12.34.193:54019)
0926 15:09:38.667356 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0926 15:09:38.667496 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.254:42537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260926 15:09:38.678139 19375 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54126:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 15:09:38.678457 19375 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54126:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 15:09:38.684010 19375 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 70d9c6c1008544b4b7ba623a53d9e680: Sending DeleteTablet for 3 replicas of tablet a273bedf1df042cdb7546e04fcbdaeff
I20260926 15:09:38.685003 19778 tablet_service.cc:1514] Processing DeleteTablet for tablet a273bedf1df042cdb7546e04fcbdaeff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:09:38 UTC) from {username='oryx', principal='oryx/127.12.34.252@KRBTEST.COM'} at 127.0.0.1:38926
I20260926 15:09:38.685217 19637 tablet_service.cc:1514] Processing DeleteTablet for tablet a273bedf1df042cdb7546e04fcbdaeff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:09:38 UTC) from {username='oryx', principal='oryx/127.12.34.252@KRBTEST.COM'} at 127.0.0.1:40480
I20260926 15:09:38.685460 19489 tablet_service.cc:1514] Processing DeleteTablet for tablet a273bedf1df042cdb7546e04fcbdaeff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:09:38 UTC) from {username='oryx', principal='oryx/127.12.34.252@KRBTEST.COM'} at 127.0.0.1:57018
I20260926 15:09:38.685686 19962 tablet_replica.cc:330] T a273bedf1df042cdb7546e04fcbdaeff P 71e96bd442d34587b6d0e98bb1c8aae4: stopping tablet replica
I20260926 15:09:38.685846 19962 raft_consensus.cc:2229] T a273bedf1df042cdb7546e04fcbdaeff P 71e96bd442d34587b6d0e98bb1c8aae4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 15:09:38.686149 19962 raft_consensus.cc:2258] T a273bedf1df042cdb7546e04fcbdaeff P 71e96bd442d34587b6d0e98bb1c8aae4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:09:38.686509 12427 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 19356
I20260926 15:09:38.687474 19962 ts_tablet_manager.cc:1905] T a273bedf1df042cdb7546e04fcbdaeff P 71e96bd442d34587b6d0e98bb1c8aae4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:09:38.696295 19962 ts_tablet_manager.cc:1918] T a273bedf1df042cdb7546e04fcbdaeff P 71e96bd442d34587b6d0e98bb1c8aae4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 15:09:38.696597 19962 log.cc:1196] T a273bedf1df042cdb7546e04fcbdaeff P 71e96bd442d34587b6d0e98bb1c8aae4: Deleting WAL directory at /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/minicluster-data/ts-2/wal/wals/a273bedf1df042cdb7546e04fcbdaeff
I20260926 15:09:38.697111 19964 tablet_replica.cc:330] T a273bedf1df042cdb7546e04fcbdaeff P d3e156f17705476c90ae1f60578daa75: stopping tablet replica
I20260926 15:09:38.697150 19962 ts_tablet_manager.cc:1939] T a273bedf1df042cdb7546e04fcbdaeff P 71e96bd442d34587b6d0e98bb1c8aae4: Deleting consensus metadata
I20260926 15:09:38.697311 19964 raft_consensus.cc:2229] T a273bedf1df042cdb7546e04fcbdaeff P d3e156f17705476c90ae1f60578daa75 [term 1 LEADER]: Raft consensus shutting down.
I20260926 15:09:38.697873 19964 raft_consensus.cc:2258] T a273bedf1df042cdb7546e04fcbdaeff P d3e156f17705476c90ae1f60578daa75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:09:38.698536 19360 catalog_manager.cc:4841] TS 71e96bd442d34587b6d0e98bb1c8aae4 (127.12.34.195:36967): tablet a273bedf1df042cdb7546e04fcbdaeff (table test-table [id=8c3b946f89304340b815c70025d3f900]) successfully deleted
I20260926 15:09:38.699330 19964 ts_tablet_manager.cc:1905] T a273bedf1df042cdb7546e04fcbdaeff P d3e156f17705476c90ae1f60578daa75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:09:38.704054 19963 tablet_replica.cc:330] T a273bedf1df042cdb7546e04fcbdaeff P 85dd0480684b43909497b35c05e43908: stopping tablet replica
I20260926 15:09:38.704252 19963 raft_consensus.cc:2229] T a273bedf1df042cdb7546e04fcbdaeff P 85dd0480684b43909497b35c05e43908 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 15:09:38.704556 19963 raft_consensus.cc:2258] T a273bedf1df042cdb7546e04fcbdaeff P 85dd0480684b43909497b35c05e43908 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:09:38.705585 19963 ts_tablet_manager.cc:1905] T a273bedf1df042cdb7546e04fcbdaeff P 85dd0480684b43909497b35c05e43908: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:09:38.706239 19964 ts_tablet_manager.cc:1918] T a273bedf1df042cdb7546e04fcbdaeff P d3e156f17705476c90ae1f60578daa75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 15:09:38.706411 19964 log.cc:1196] T a273bedf1df042cdb7546e04fcbdaeff P d3e156f17705476c90ae1f60578daa75: Deleting WAL directory at /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/minicluster-data/ts-0/wal/wals/a273bedf1df042cdb7546e04fcbdaeff
I20260926 15:09:38.706907 19964 ts_tablet_manager.cc:1939] T a273bedf1df042cdb7546e04fcbdaeff P d3e156f17705476c90ae1f60578daa75: Deleting consensus metadata
I20260926 15:09:38.708036 19359 catalog_manager.cc:4841] TS d3e156f17705476c90ae1f60578daa75 (127.12.34.193:38979): tablet a273bedf1df042cdb7546e04fcbdaeff (table test-table [id=8c3b946f89304340b815c70025d3f900]) successfully deleted
I20260926 15:09:38.710686 19963 ts_tablet_manager.cc:1918] T a273bedf1df042cdb7546e04fcbdaeff P 85dd0480684b43909497b35c05e43908: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 15:09:38.710829 19963 log.cc:1196] T a273bedf1df042cdb7546e04fcbdaeff P 85dd0480684b43909497b35c05e43908: Deleting WAL directory at /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790435318278681-12427-0/minicluster-data/ts-1/wal/wals/a273bedf1df042cdb7546e04fcbdaeff
I20260926 15:09:38.711251 19963 ts_tablet_manager.cc:1939] T a273bedf1df042cdb7546e04fcbdaeff P 85dd0480684b43909497b35c05e43908: Deleting consensus metadata
I20260926 15:09:38.712622 19359 catalog_manager.cc:4841] TS 85dd0480684b43909497b35c05e43908 (127.12.34.194:38821): tablet a273bedf1df042cdb7546e04fcbdaeff (table test-table [id=8c3b946f89304340b815c70025d3f900]) successfully deleted
I20260926 15:09:38.717470 19935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:38.688350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57048 (local address 127.12.34.193:38979)
0926 15:09:38.688560 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:38.688569 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:38.700414 (+ 11845us) server_negotiation.cc:407] Connection header received
0926 15:09:38.700624 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:38.700644 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:38.700769 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:38.700896 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:09:38.703469 (+ 2573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:38.704760 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:38.706125 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:38.706788 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:38.706915 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:38.710462 (+ 3547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:09:38.710475 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:09:38.710493 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:09:38.710543 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:09:38.714550 (+ 4007us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:38.715203 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:38.715211 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:38.715215 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:38.715407 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:38.715844 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:38.715849 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:38.715853 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:38.716254 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:09:38.716554 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0926 15:09:38.716841 (+ 287us) server_negotiation.cc:299] Negotiation successful
0926 15:09:38.717050 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 15:09:38.718190 19549 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:09:38.762176 19934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:38.761737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:42537 (local address 127.0.0.1:37356)
0926 15:09:38.761918 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0926 15:09:38.762024 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.254:42537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260926 15:09:38.763237 19934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:38.762862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:42537 (local address 127.0.0.1:37362)
0926 15:09:38.763011 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0926 15:09:38.763097 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.254:42537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260926 15:09:38.819504 12427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8uoJGv/build/asan/bin/kudu with pid 19356
I20260926 15:09:38.838338 12427 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 19566
I20260926 15:09:38.858346 19932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:38.839012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40488 (local address 127.12.34.194:38821)
0926 15:09:38.839170 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:38.839174 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:38.839215 (+ 41us) server_negotiation.cc:407] Connection header received
0926 15:09:38.839393 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:38.839397 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:38.839459 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:38.839543 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:09:38.840783 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:38.841625 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:38.842823 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:38.843219 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:38.843316 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:38.853159 (+ 9843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:09:38.853174 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:09:38.853192 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:09:38.853260 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:09:38.856052 (+ 2792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:38.856611 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:38.856617 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:38.856619 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:38.856732 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:38.857034 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:38.857037 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:38.857039 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:38.857439 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:09:38.857635 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0926 15:09:38.857879 (+ 244us) server_negotiation.cc:299] Negotiation successful
0926 15:09:38.858072 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 15:09:38.858955 19697 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:09:38.975602 12427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8uoJGv/build/asan/bin/kudu with pid 19566
I20260926 15:09:38.997061 12427 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 19715
I20260926 15:09:39.013175 19933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:38.997881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38928 (local address 127.12.34.195:36967)
0926 15:09:38.998044 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:38.998048 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:38.998085 (+ 37us) server_negotiation.cc:407] Connection header received
0926 15:09:38.998255 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:38.998259 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:38.998355 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:38.998440 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:09:38.999844 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:39.000737 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:39.002470 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:39.002876 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:39.002974 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:39.006355 (+ 3381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:09:39.006375 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:09:39.006399 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:09:39.006450 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:09:39.010094 (+ 3644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:39.010928 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:39.010936 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:39.010939 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:39.011079 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:39.011483 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:39.011487 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:39.011489 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:39.011850 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:09:39.012094 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0926 15:09:39.012451 (+ 357us) server_negotiation.cc:299] Negotiation successful
0926 15:09:39.012604 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 15:09:39.013837 19838 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:09:39.108860 12427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8uoJGv/build/asan/bin/kudu with pid 19715
I20260926 15:09:39.126535 12427 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 19049
I20260926 15:09:39.127671 19217 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:09:39.244807 12427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8uoJGv/build/asan/bin/kudu with pid 19049
W20260926 15:09:39.265139 19362 connection.cc:535] client connection to 127.12.34.253:32783 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 15:09:39.265607 12427 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 19150
I20260926 15:09:39.266782 19405 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:09:39.384945 19971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:39.266098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.253:32783 (local address 127.0.0.1:34624)
0926 15:09:39.266596 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0926 15:09:39.266713 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.253:32783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":225,"threads_started":1}
W20260926 15:09:39.385272 19362 consensus_peers.cc:487] T 00000000000000000000000000000000 P 70d9c6c1008544b4b7ba623a53d9e680 -> Peer f183e2953d3141608ce600f9167cd229 (127.12.34.253:32783): Couldn't send request to peer f183e2953d3141608ce600f9167cd229. Status: Network error: Client connection negotiation failed: client connection to 127.12.34.253:32783: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260926 15:09:39.385342 12427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8uoJGv/build/asan/bin/kudu with pid 19150
2026-09-26T15:09:39Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (9532 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435318278681-12427-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 15:09:39 dist-test-slave-865v krb5kdc[19976](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 15:09:39 dist-test-slave-865v krb5kdc[19976](info): set up 2 sockets
Sep 26 15:09:39 dist-test-slave-865v krb5kdc[19976](info): commencing operation
krb5kdc: starting...
W20260926 15:09:40.514154 12427 mini_kdc.cc:121] Time spent starting KDC: real 1.084s user 0.001s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435318278681-12427-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435318278681-12427-0/krb5kdc/test-user.keytab.
Sep 26 15:09:40 dist-test-slave-865v krb5kdc[19976](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435380, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T15:09:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T15:09:40Z Disabled control of system clock
WARNING: no policy specified for oryx/127.12.34.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.34.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.34.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435318278681-12427-0/krb5kdc/oryx_127.12.34.254.keytab.
Entry for principal oryx/127.12.34.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435318278681-12427-0/krb5kdc/oryx_127.12.34.254.keytab.
I20260926 15:09:40.728829 12427 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8uoJGv/build/asan/bin/kudu
/tmp/dist-test-task8uoJGv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435318278681-12427-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435318278681-12427-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435318278681-12427-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.34.254:46481
--webserver_interface=127.12.34.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435318278681-12427-0/krb5kdc/oryx_127.12.34.254.keytab
--principal=oryx/127.12.34.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.34.212:42113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.34.254:46481
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435318278681-12427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435318278681-12427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435318278681-12427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:09:41.055379 19989 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:09:41.055631 19989 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:09:41.055776 19989 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:09:41.062687 19989 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:09:41.062772 19989 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 15:09:41.062875 19989 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:09:41.062963 19989 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:09:41.063035 19989 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:09:41.063145 19989 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:09:41.072952 19989 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.34.212:42113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435318278681-12427-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435318278681-12427-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.34.254:46481
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435318278681-12427-0/krb5kdc/oryx_127.12.34.254.keytab
--principal=oryx/127.12.34.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.34.254:46481
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.34.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435318278681-12427-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:09:41.074115 19989 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:09:41.075349 19989 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:09:41.086675 19995 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:09:41.086886 19994 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:09:41.086943 19997 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:09:41.087913 19989 server_base.cc:1034] running on GCE node
I20260926 15:09:41.088582 19989 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:09:41.089902 19989 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:09:41.091138 19989 hybrid_clock.cc:648] HybridClock initialized: now 1790435381091064 us; error 38 us; skew 500 ppm
I20260926 15:09:41.091485 19989 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:09:41 dist-test-slave-865v krb5kdc[19976](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435381, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.34.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:09:41.094630 19989 init.cc:377] Logged in from keytab as oryx/127.12.34.254@KRBTEST.COM (short username oryx)
I20260926 15:09:41.096025 19989 webserver.cc:460] Webserver started at http://127.12.34.254:34821/ using document root <none> and password file <none>
I20260926 15:09:41.096618 19989 fs_manager.cc:359] Metadata directory not provided
I20260926 15:09:41.096699 19989 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:09:41.096961 19989 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:09:41.099692 19989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435318278681-12427-0/minicluster-data/master-0/data/instance:
uuid: "1a3d30fc883a4f67bc16c22629712a96"
format_stamp: "Formatted at 2026-09-26 15:09:41 on dist-test-slave-865v"
server_key: "32df8c642233213d866b93f150112810"
server_key_iv: "24d99daf109f7c81f44ddc3a768d38c4"
server_key_version: "encryptionkey@0"
I20260926 15:09:41.100466 19989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435318278681-12427-0/minicluster-data/master-0/wal/instance:
uuid: "1a3d30fc883a4f67bc16c22629712a96"
format_stamp: "Formatted at 2026-09-26 15:09:41 on dist-test-slave-865v"
server_key: "32df8c642233213d866b93f150112810"
server_key_iv: "24d99daf109f7c81f44ddc3a768d38c4"
server_key_version: "encryptionkey@0"
I20260926 15:09:41.105063 19989 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.002s
I20260926 15:09:41.108342 20004 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:09:41.109586 19989 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260926 15:09:41.109740 19989 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435318278681-12427-0/minicluster-data/master-0/data,/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435318278681-12427-0/minicluster-data/master-0/wal
uuid: "1a3d30fc883a4f67bc16c22629712a96"
format_stamp: "Formatted at 2026-09-26 15:09:41 on dist-test-slave-865v"
server_key: "32df8c642233213d866b93f150112810"
server_key_iv: "24d99daf109f7c81f44ddc3a768d38c4"
server_key_version: "encryptionkey@0"
I20260926 15:09:41.109920 19989 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435318278681-12427-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435318278681-12427-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435318278681-12427-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:09:41.123168 19989 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:09:41.127055 19989 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:09:41.127327 19989 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:09:41.148590 19989 rpc_server.cc:307] RPC server started. Bound to: 127.12.34.254:46481
I20260926 15:09:41.148612 20065 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.34.254:46481 every 8 connection(s)
I20260926 15:09:41.149874 19989 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
I20260926 15:09:41.153458 20066 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:09:41.158380 12427 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8uoJGv/build/asan/bin/kudu as pid 19989
I20260926 15:09:41.158573 12427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435318278681-12427-0/minicluster-data/master-0/wal/instance
I20260926 15:09:41.159354 12427 external_mini_cluster.cc:1428] Setting key 18f5a64e08190b17ac41b9db7a3b023a
I20260926 15:09:41.165884 20066 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a3d30fc883a4f67bc16c22629712a96: Bootstrap starting.
I20260926 15:09:41.169369 20066 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1a3d30fc883a4f67bc16c22629712a96: Neither blocks nor log segments found. Creating new log.
I20260926 15:09:41.170532 20066 log.cc:824] T 00000000000000000000000000000000 P 1a3d30fc883a4f67bc16c22629712a96: Log is configured to *not* fsync() on all Append() calls
I20260926 15:09:41.173766 20066 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a3d30fc883a4f67bc16c22629712a96: No bootstrap required, opened a new log
Sep 26 15:09:41 dist-test-slave-865v krb5kdc[19976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435380, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.12.34.254@KRBTEST.COM
I20260926 15:09:41.184314 20069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:41.161380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43052 (local address 127.12.34.254:46481)
0926 15:09:41.162430 (+ 1050us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:41.162450 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:41.162501 (+ 51us) server_negotiation.cc:407] Connection header received
0926 15:09:41.165187 (+ 2686us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:41.165213 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:41.165633 (+ 420us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:41.165926 (+ 293us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:09:41.167595 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:41.168810 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:41.170679 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:41.171701 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:41.171867 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:41.176197 (+ 4330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:09:41.176207 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:09:41.176226 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:09:41.176267 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:09:41.180113 (+ 3846us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:41.180754 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:41.180762 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:41.180769 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:41.180966 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:41.181358 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:41.181363 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:41.181367 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:41.182056 (+ 689us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:09:41.182349 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0926 15:09:41.182651 (+ 302us) server_negotiation.cc:299] Negotiation successful
0926 15:09:41.183076 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":573,"thread_start_us":350,"threads_started":1}
I20260926 15:09:41.184074 20066 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1a3d30fc883a4f67bc16c22629712a96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a3d30fc883a4f67bc16c22629712a96" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 46481 } }
I20260926 15:09:41.184486 20066 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1a3d30fc883a4f67bc16c22629712a96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:09:41.184619 20066 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1a3d30fc883a4f67bc16c22629712a96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a3d30fc883a4f67bc16c22629712a96, State: Initialized, Role: FOLLOWER
I20260926 15:09:41.185267 20066 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1a3d30fc883a4f67bc16c22629712a96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a3d30fc883a4f67bc16c22629712a96" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 46481 } }
I20260926 15:09:41.185508 20066 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1a3d30fc883a4f67bc16c22629712a96 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 15:09:41.185655 20066 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1a3d30fc883a4f67bc16c22629712a96 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 15:09:41.185851 20066 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1a3d30fc883a4f67bc16c22629712a96 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:09:41.188021 20066 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1a3d30fc883a4f67bc16c22629712a96 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a3d30fc883a4f67bc16c22629712a96" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 46481 } }
I20260926 15:09:41.188480 20066 leader_election.cc:302] T 00000000000000000000000000000000 P 1a3d30fc883a4f67bc16c22629712a96 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1a3d30fc883a4f67bc16c22629712a96; no voters:
I20260926 15:09:41.189003 20066 leader_election.cc:288] T 00000000000000000000000000000000 P 1a3d30fc883a4f67bc16c22629712a96 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 15:09:41.189157 20071 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1a3d30fc883a4f67bc16c22629712a96 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:09:41.189880 20071 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1a3d30fc883a4f67bc16c22629712a96 [term 1 LEADER]: Becoming Leader. State: Replica: 1a3d30fc883a4f67bc16c22629712a96, State: Running, Role: LEADER
I20260926 15:09:41.190442 20071 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1a3d30fc883a4f67bc16c22629712a96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a3d30fc883a4f67bc16c22629712a96" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 46481 } }
I20260926 15:09:41.191083 20066 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1a3d30fc883a4f67bc16c22629712a96 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:09:41.193312 20073 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a3d30fc883a4f67bc16c22629712a96 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1a3d30fc883a4f67bc16c22629712a96. Latest consensus state: current_term: 1 leader_uuid: "1a3d30fc883a4f67bc16c22629712a96" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a3d30fc883a4f67bc16c22629712a96" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 46481 } } }
I20260926 15:09:41.193668 20073 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a3d30fc883a4f67bc16c22629712a96 [sys.catalog]: This master's current role is: LEADER
I20260926 15:09:41.193421 20072 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a3d30fc883a4f67bc16c22629712a96 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1a3d30fc883a4f67bc16c22629712a96" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a3d30fc883a4f67bc16c22629712a96" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 46481 } } }
I20260926 15:09:41.193782 20072 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a3d30fc883a4f67bc16c22629712a96 [sys.catalog]: This master's current role is: LEADER
I20260926 15:09:41.197885 20077 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:09:41.204058 20077 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 15:09:41.213806 20077 catalog_manager.cc:1347] Generated new cluster ID: 8389060bdbca4268bb47bd88a941d0ec
I20260926 15:09:41.213943 20077 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 15:09:41.251560 20077 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 15:09:41.252883 20077 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:09:41.266413 20077 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1a3d30fc883a4f67bc16c22629712a96: Generated new TSK 0
I20260926 15:09:41.267185 20077 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.12.34.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.34.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.34.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435318278681-12427-0/krb5kdc/oryx_127.12.34.193.keytab.
Entry for principal oryx/127.12.34.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435318278681-12427-0/krb5kdc/oryx_127.12.34.193.keytab.
I20260926 15:09:41.321189 12427 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8uoJGv/build/asan/bin/kudu
/tmp/dist-test-task8uoJGv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435318278681-12427-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435318278681-12427-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435318278681-12427-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435318278681-12427-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.34.193:0
--local_ip_for_outbound_sockets=127.12.34.193
--webserver_interface=127.12.34.193
--webserver_port=0
--tserver_master_addrs=127.12.34.254:46481
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435318278681-12427-0/krb5kdc/oryx_127.12.34.193.keytab
--principal=oryx/127.12.34.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.34.212:42113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435318278681-12427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435318278681-12427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435318278681-12427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:09:41.640054 20092 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:09:41.640303 20092 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:09:41.640458 20092 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:09:41.647460 20092 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:09:41.647549 20092 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:09:41.647722 20092 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 15:09:41.647848 20092 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.34.193
I20260926 15:09:41.657016 20092 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.34.212:42113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435318278681-12427-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435318278681-12427-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435318278681-12427-0/krb5kdc/oryx_127.12.34.193.keytab
--principal=oryx/127.12.34.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.34.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435318278681-12427-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.34.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.12.34.254:46481
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.34.193
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435318278681-12427-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:09:41.658198 20092 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:09:41.659493 20092 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:09:41.671794 20098 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:09:41.671834 20097 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:09:41.671875 20100 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:09:41.672581 20092 server_base.cc:1034] running on GCE node
I20260926 15:09:41.673193 20092 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:09:41.674094 20092 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:09:41.675292 20092 hybrid_clock.cc:648] HybridClock initialized: now 1790435381675217 us; error 58 us; skew 500 ppm
I20260926 15:09:41.675721 20092 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:09:41 dist-test-slave-865v krb5kdc[19976](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435381, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.34.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:09:41.678823 20092 init.cc:377] Logged in from keytab as oryx/127.12.34.193@KRBTEST.COM (short username oryx)
I20260926 15:09:41.680135 20092 webserver.cc:460] Webserver started at http://127.12.34.193:41061/ using document root <none> and password file <none>
I20260926 15:09:41.680819 20092 fs_manager.cc:359] Metadata directory not provided
I20260926 15:09:41.680949 20092 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:09:41.681317 20092 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:09:41.684338 20092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435318278681-12427-0/minicluster-data/ts-0/data/instance:
uuid: "bffb811d93894ea5a14e6f4f56f4a196"
format_stamp: "Formatted at 2026-09-26 15:09:41 on dist-test-slave-865v"
server_key: "89062389916814797f58523840643ba2"
server_key_iv: "bd85515eb8dec1a99d23caf12f1d8e4e"
server_key_version: "encryptionkey@0"
I20260926 15:09:41.685302 20092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435318278681-12427-0/minicluster-data/ts-0/wal/instance:
uuid: "bffb811d93894ea5a14e6f4f56f4a196"
format_stamp: "Formatted at 2026-09-26 15:09:41 on dist-test-slave-865v"
server_key: "89062389916814797f58523840643ba2"
server_key_iv: "bd85515eb8dec1a99d23caf12f1d8e4e"
server_key_version: "encryptionkey@0"
I20260926 15:09:41.690397 20092 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260926 15:09:41.694010 20107 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:09:41.695549 20092 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 15:09:41.695715 20092 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435318278681-12427-0/minicluster-data/ts-0/data,/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435318278681-12427-0/minicluster-data/ts-0/wal
uuid: "bffb811d93894ea5a14e6f4f56f4a196"
format_stamp: "Formatted at 2026-09-26 15:09:41 on dist-test-slave-865v"
server_key: "89062389916814797f58523840643ba2"
server_key_iv: "bd85515eb8dec1a99d23caf12f1d8e4e"
server_key_version: "encryptionkey@0"
I20260926 15:09:41.695933 20092 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435318278681-12427-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435318278681-12427-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435318278681-12427-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:09:41.717723 20092 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:09:41.721661 20092 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:09:41.721925 20092 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:09:41.724542 20092 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:09:41.724659 20092 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:09:41.724781 20092 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:09:41.724850 20092 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 15:09:41 dist-test-slave-865v krb5kdc[19976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435381, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.34.193@KRBTEST.COM for oryx/127.12.34.254@KRBTEST.COM
I20260926 15:09:41.751261 20117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:41.726007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:46481 (local address 127.12.34.193:40531)
0926 15:09:41.726762 (+ 755us) negotiation.cc:107] Waiting for socket to connect
0926 15:09:41.726807 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 15:09:41.728772 (+ 1965us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:09:41.729784 (+ 1012us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:09:41.729800 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:09:41.730762 (+ 962us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:09:41.731948 (+ 1186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:41.731978 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:41.733317 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:41.733323 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:41.736098 (+ 2775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:41.736119 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:41.737336 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:41.737347 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:41.737642 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:41.738344 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:09:41.738389 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:09:41.741611 (+ 3222us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:09:41.745960 (+ 4349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:09:41.745973 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:09:41.745997 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:09:41.746797 (+ 800us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:09:41.747246 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:09:41.747251 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:09:41.747255 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:09:41.747568 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:09:41.748385 (+ 817us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:09:41.748391 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:09:41.748856 (+ 465us) client_negotiation.cc:769] Sending connection context
0926 15:09:41.749266 (+ 410us) client_negotiation.cc:240] Negotiation successful
0926 15:09:41.749664 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":330,"threads_started":1}
I20260926 15:09:41.752182 20116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:41.725455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.193:40531 (local address 127.12.34.254:46481)
0926 15:09:41.725939 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:41.725944 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:41.726922 (+ 978us) server_negotiation.cc:407] Connection header received
0926 15:09:41.729101 (+ 2179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:41.729107 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:41.729193 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:41.729324 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:09:41.732167 (+ 2843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:41.733165 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:41.736292 (+ 3127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:41.736853 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:41.736981 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:41.741837 (+ 4856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:09:41.741842 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:09:41.741846 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:09:41.741876 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:09:41.745763 (+ 3887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:41.746949 (+ 1186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:41.746956 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:41.746962 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:41.747115 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:41.747722 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:41.747730 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:41.747736 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:41.748206 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:09:41.748383 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0926 15:09:41.751779 (+ 3396us) server_negotiation.cc:299] Negotiation successful
0926 15:09:41.751952 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":397,"thread_start_us":266,"threads_started":1}
I20260926 15:09:41.766788 20092 rpc_server.cc:307] RPC server started. Bound to: 127.12.34.193:44487
I20260926 15:09:41.766726 20227 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.34.193:44487 every 8 connection(s)
I20260926 15:09:41.767961 20092 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435318278681-12427-0/minicluster-data/ts-0/data/info.pb
I20260926 15:09:41.770210 20228 heartbeater.cc:347] Connected to a master server at 127.12.34.254:46481
I20260926 15:09:41.770428 20228 heartbeater.cc:464] Registering TS with master...
I20260926 15:09:41.771196 20228 heartbeater.cc:511] Master 127.12.34.254:46481 requested a full tablet report, sending...
I20260926 15:09:41.773160 12427 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8uoJGv/build/asan/bin/kudu as pid 20092
I20260926 15:09:41.773341 12427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435318278681-12427-0/minicluster-data/ts-0/wal/instance
I20260926 15:09:41.773325 20021 ts_manager.cc:194] Registered new tserver with Master: bffb811d93894ea5a14e6f4f56f4a196 (127.12.34.193:44487)
I20260926 15:09:41.774091 12427 external_mini_cluster.cc:1428] Setting key a32c09a3bb423e53557278126a4e1188
I20260926 15:09:41.775246 20021 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.12.34.193@KRBTEST.COM'} at 127.12.34.193:40531
I20260926 15:09:41.775437 20116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:41.759418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.193:44253 (local address 127.12.34.254:46481)
0926 15:09:41.759623 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:41.759630 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:41.761851 (+ 2221us) server_negotiation.cc:407] Connection header received
0926 15:09:41.762095 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:41.762099 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:41.762173 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:41.762348 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:09:41.763790 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:41.764796 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:41.766522 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:41.766980 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:41.767122 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:41.769444 (+ 2322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:09:41.769449 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:09:41.769452 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:09:41.769479 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:09:41.772399 (+ 2920us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:41.773294 (+ 895us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:41.773301 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:41.773306 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:41.773498 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:41.773926 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:41.773930 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:41.773933 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:41.774284 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:09:41.774439 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0926 15:09:41.774879 (+ 440us) server_negotiation.cc:299] Negotiation successful
0926 15:09:41.775074 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":67}
I20260926 15:09:41.776368 20208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:41.758852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:46481 (local address 127.12.34.193:44253)
0926 15:09:41.761728 (+ 2876us) negotiation.cc:107] Waiting for socket to connect
0926 15:09:41.761753 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 15:09:41.761933 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:09:41.762365 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:09:41.762370 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:09:41.763107 (+ 737us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:09:41.763572 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:41.763592 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:41.764926 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:41.764933 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:41.766353 (+ 1420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:41.766365 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:41.767164 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:41.767170 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:41.767361 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:41.768016 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:09:41.768033 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:09:41.769172 (+ 1139us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:09:41.772542 (+ 3370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:09:41.772546 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:09:41.772549 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:09:41.773151 (+ 602us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:09:41.773629 (+ 478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:09:41.773634 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:09:41.773638 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:09:41.773789 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:09:41.774417 (+ 628us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:09:41.774422 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:09:41.774603 (+ 181us) client_negotiation.cc:769] Sending connection context
0926 15:09:41.775641 (+ 1038us) client_negotiation.cc:240] Negotiation successful
0926 15:09:41.775925 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2685,"thread_start_us":2461,"threads_started":1}
WARNING: no policy specified for oryx/127.12.34.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.34.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.34.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435318278681-12427-0/krb5kdc/oryx_127.12.34.194.keytab.
Entry for principal oryx/127.12.34.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435318278681-12427-0/krb5kdc/oryx_127.12.34.194.keytab.
I20260926 15:09:41.821766 12427 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8uoJGv/build/asan/bin/kudu
/tmp/dist-test-task8uoJGv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435318278681-12427-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435318278681-12427-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435318278681-12427-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435318278681-12427-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.34.194:0
--local_ip_for_outbound_sockets=127.12.34.194
--webserver_interface=127.12.34.194
--webserver_port=0
--tserver_master_addrs=127.12.34.254:46481
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435318278681-12427-0/krb5kdc/oryx_127.12.34.194.keytab
--principal=oryx/127.12.34.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.34.212:42113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435318278681-12427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435318278681-12427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435318278681-12427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:09:42.133566 20232 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:09:42.133872 20232 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:09:42.134052 20232 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:09:42.141232 20232 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:09:42.141376 20232 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:09:42.141536 20232 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 15:09:42.141680 20232 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.34.194
I20260926 15:09:42.153699 20232 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.34.212:42113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435318278681-12427-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435318278681-12427-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435318278681-12427-0/krb5kdc/oryx_127.12.34.194.keytab
--principal=oryx/127.12.34.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.34.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435318278681-12427-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.34.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.12.34.254:46481
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.34.194
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435318278681-12427-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:09:42.155140 20232 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:09:42.156652 20232 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:09:42.171949 20237 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:09:42.171876 20240 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:09:42.171921 20238 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:09:42.172892 20232 server_base.cc:1034] running on GCE node
I20260926 15:09:42.173592 20232 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:09:42.174500 20232 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:09:42.175750 20232 hybrid_clock.cc:648] HybridClock initialized: now 1790435382175687 us; error 43 us; skew 500 ppm
I20260926 15:09:42.176120 20232 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:09:42 dist-test-slave-865v krb5kdc[19976](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435382, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.34.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:09:42.179296 20232 init.cc:377] Logged in from keytab as oryx/127.12.34.194@KRBTEST.COM (short username oryx)
I20260926 15:09:42.180778 20232 webserver.cc:460] Webserver started at http://127.12.34.194:38161/ using document root <none> and password file <none>
I20260926 15:09:42.181519 20232 fs_manager.cc:359] Metadata directory not provided
I20260926 15:09:42.181646 20232 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:09:42.181967 20232 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:09:42.184775 20232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435318278681-12427-0/minicluster-data/ts-1/data/instance:
uuid: "641358d46138484794ffdf3e0ceb4109"
format_stamp: "Formatted at 2026-09-26 15:09:42 on dist-test-slave-865v"
server_key: "0deb0db7d775c8e73df8a9d58b37d66d"
server_key_iv: "0c44bd13fee5df1c073673e668bb4a7a"
server_key_version: "encryptionkey@0"
I20260926 15:09:42.185623 20232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435318278681-12427-0/minicluster-data/ts-1/wal/instance:
uuid: "641358d46138484794ffdf3e0ceb4109"
format_stamp: "Formatted at 2026-09-26 15:09:42 on dist-test-slave-865v"
server_key: "0deb0db7d775c8e73df8a9d58b37d66d"
server_key_iv: "0c44bd13fee5df1c073673e668bb4a7a"
server_key_version: "encryptionkey@0"
I20260926 15:09:42.190918 20232 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 15:09:42.194511 20247 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:09:42.195771 20232 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 15:09:42.195900 20232 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435318278681-12427-0/minicluster-data/ts-1/data,/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435318278681-12427-0/minicluster-data/ts-1/wal
uuid: "641358d46138484794ffdf3e0ceb4109"
format_stamp: "Formatted at 2026-09-26 15:09:42 on dist-test-slave-865v"
server_key: "0deb0db7d775c8e73df8a9d58b37d66d"
server_key_iv: "0c44bd13fee5df1c073673e668bb4a7a"
server_key_version: "encryptionkey@0"
I20260926 15:09:42.196102 20232 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435318278681-12427-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435318278681-12427-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435318278681-12427-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:09:42.213208 20232 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:09:42.217569 20232 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:09:42.217796 20232 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:09:42.220477 20232 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:09:42.220578 20232 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:09:42.220732 20232 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:09:42.220813 20232 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 15:09:42 dist-test-slave-865v krb5kdc[19976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435382, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.34.194@KRBTEST.COM for oryx/127.12.34.254@KRBTEST.COM
I20260926 15:09:42.242491 20116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:42.220978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.194:51161 (local address 127.12.34.254:46481)
0926 15:09:42.221196 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:42.221204 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:42.222478 (+ 1274us) server_negotiation.cc:407] Connection header received
0926 15:09:42.224578 (+ 2100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:42.224586 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:42.224680 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:42.224809 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:09:42.226964 (+ 2155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:42.228302 (+ 1338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:42.230979 (+ 2677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:42.231401 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:42.231542 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:42.235804 (+ 4262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:09:42.235809 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:09:42.235813 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:09:42.235844 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:09:42.239038 (+ 3194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:42.239907 (+ 869us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:42.239912 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:42.239916 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:42.240094 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:42.240611 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:42.240617 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:42.240620 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:42.240894 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:09:42.241054 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0926 15:09:42.241924 (+ 870us) server_negotiation.cc:299] Negotiation successful
0926 15:09:42.242132 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 15:09:42.243602 20256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:42.221597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:46481 (local address 127.12.34.194:51161)
0926 15:09:42.222326 (+ 729us) negotiation.cc:107] Waiting for socket to connect
0926 15:09:42.222361 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 15:09:42.224180 (+ 1819us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:09:42.225139 (+ 959us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:09:42.225152 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:09:42.225887 (+ 735us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:09:42.226698 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:42.226719 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:42.228498 (+ 1779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:42.228509 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:42.230830 (+ 2321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:42.230841 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:42.231653 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:42.231661 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:42.231867 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:42.232627 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:09:42.232670 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:09:42.235587 (+ 2917us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:09:42.239152 (+ 3565us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:09:42.239161 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:09:42.239177 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:09:42.239770 (+ 593us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:09:42.240226 (+ 456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:09:42.240231 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:09:42.240234 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:09:42.240482 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:09:42.241018 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:09:42.241027 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:09:42.241476 (+ 449us) client_negotiation.cc:769] Sending connection context
0926 15:09:42.241828 (+ 352us) client_negotiation.cc:240] Negotiation successful
0926 15:09:42.242370 (+ 542us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":336,"threads_started":1}
I20260926 15:09:42.266608 20116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:42.250843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.194:41557 (local address 127.12.34.254:46481)
0926 15:09:42.251092 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:42.251100 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:42.251404 (+ 304us) server_negotiation.cc:407] Connection header received
0926 15:09:42.251542 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:42.251550 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:42.251637 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:42.251745 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:09:42.253440 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:42.254401 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:42.255984 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:42.256453 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:42.256585 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:42.259299 (+ 2714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:09:42.259307 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:09:42.259313 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:09:42.259352 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:09:42.262887 (+ 3535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:42.264077 (+ 1190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:42.264085 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:42.264089 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:42.264259 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:42.264761 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:42.264768 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:42.264772 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:42.265164 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:09:42.265324 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0926 15:09:42.265919 (+ 595us) server_negotiation.cc:299] Negotiation successful
0926 15:09:42.266210 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260926 15:09:42.266917 20319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:42.250338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:46481 (local address 127.12.34.194:41557)
0926 15:09:42.251291 (+ 953us) negotiation.cc:107] Waiting for socket to connect
0926 15:09:42.251309 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 15:09:42.251435 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:09:42.251784 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:09:42.251787 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:09:42.252647 (+ 860us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:09:42.253272 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:42.253291 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:42.254538 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:42.254545 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:42.255840 (+ 1295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:42.255851 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:42.256637 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:42.256643 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:42.256852 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:42.257666 (+ 814us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:09:42.257697 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:09:42.259057 (+ 1360us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:09:42.263054 (+ 3997us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:09:42.263062 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:09:42.263069 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:09:42.263918 (+ 849us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:09:42.264395 (+ 477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:09:42.264403 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:09:42.264409 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:09:42.264611 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:09:42.265316 (+ 705us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:09:42.265322 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:09:42.265602 (+ 280us) client_negotiation.cc:769] Sending connection context
0926 15:09:42.265812 (+ 210us) client_negotiation.cc:240] Negotiation successful
0926 15:09:42.266277 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":804,"thread_start_us":676,"threads_started":1}
I20260926 15:09:42.270538 20232 rpc_server.cc:307] RPC server started. Bound to: 127.12.34.194:35581
I20260926 15:09:42.270594 20366 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.34.194:35581 every 8 connection(s)
I20260926 15:09:42.271862 20232 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435318278681-12427-0/minicluster-data/ts-1/data/info.pb
I20260926 15:09:42.273779 20367 heartbeater.cc:347] Connected to a master server at 127.12.34.254:46481
I20260926 15:09:42.274001 20367 heartbeater.cc:464] Registering TS with master...
I20260926 15:09:42.274590 20367 heartbeater.cc:511] Master 127.12.34.254:46481 requested a full tablet report, sending...
I20260926 15:09:42.275949 20021 ts_manager.cc:194] Registered new tserver with Master: 641358d46138484794ffdf3e0ceb4109 (127.12.34.194:35581)
I20260926 15:09:42.276651 20021 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.12.34.194@KRBTEST.COM'} at 127.12.34.194:51161
I20260926 15:09:42.281407 12427 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8uoJGv/build/asan/bin/kudu as pid 20232
I20260926 15:09:42.281579 12427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435318278681-12427-0/minicluster-data/ts-1/wal/instance
I20260926 15:09:42.282019 12427 external_mini_cluster.cc:1428] Setting key 27c1279dfd5fe2cd17d283ffa11dfc47
WARNING: no policy specified for oryx/127.12.34.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.34.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.34.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435318278681-12427-0/krb5kdc/oryx_127.12.34.195.keytab.
Entry for principal oryx/127.12.34.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435318278681-12427-0/krb5kdc/oryx_127.12.34.195.keytab.
I20260926 15:09:42.329793 12427 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8uoJGv/build/asan/bin/kudu
/tmp/dist-test-task8uoJGv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435318278681-12427-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435318278681-12427-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435318278681-12427-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435318278681-12427-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.34.195:0
--local_ip_for_outbound_sockets=127.12.34.195
--webserver_interface=127.12.34.195
--webserver_port=0
--tserver_master_addrs=127.12.34.254:46481
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435318278681-12427-0/krb5kdc/oryx_127.12.34.195.keytab
--principal=oryx/127.12.34.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.34.212:42113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435318278681-12427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435318278681-12427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435318278681-12427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:09:42.646579 20371 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:09:42.646865 20371 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:09:42.647025 20371 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:09:42.654131 20371 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:09:42.654261 20371 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:09:42.654398 20371 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 15:09:42.654531 20371 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.34.195
I20260926 15:09:42.663829 20371 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.34.212:42113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435318278681-12427-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435318278681-12427-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435318278681-12427-0/krb5kdc/oryx_127.12.34.195.keytab
--principal=oryx/127.12.34.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.34.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435318278681-12427-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.34.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.12.34.254:46481
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.34.195
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435318278681-12427-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:09:42.665014 20371 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:09:42.666270 20371 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:09:42.679430 20379 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:09:42.679507 20377 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:09:42.679482 20376 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:09:42.680091 20371 server_base.cc:1034] running on GCE node
I20260926 15:09:42.680806 20371 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:09:42.682403 20371 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:09:42.683669 20371 hybrid_clock.cc:648] HybridClock initialized: now 1790435382683590 us; error 52 us; skew 500 ppm
I20260926 15:09:42.683985 20371 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:09:42 dist-test-slave-865v krb5kdc[19976](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435382, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.34.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:09:42.687178 20371 init.cc:377] Logged in from keytab as oryx/127.12.34.195@KRBTEST.COM (short username oryx)
I20260926 15:09:42.688769 20371 webserver.cc:460] Webserver started at http://127.12.34.195:36649/ using document root <none> and password file <none>
I20260926 15:09:42.689500 20371 fs_manager.cc:359] Metadata directory not provided
I20260926 15:09:42.689625 20371 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:09:42.689929 20371 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:09:42.692744 20371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435318278681-12427-0/minicluster-data/ts-2/data/instance:
uuid: "3793e3bed5e04acb8adbc620baf660a4"
format_stamp: "Formatted at 2026-09-26 15:09:42 on dist-test-slave-865v"
server_key: "a880f1e4c63facdf054cd64a0f42df94"
server_key_iv: "5f2b986b7894dfa696e698a4ddbf3085"
server_key_version: "encryptionkey@0"
I20260926 15:09:42.693650 20371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435318278681-12427-0/minicluster-data/ts-2/wal/instance:
uuid: "3793e3bed5e04acb8adbc620baf660a4"
format_stamp: "Formatted at 2026-09-26 15:09:42 on dist-test-slave-865v"
server_key: "a880f1e4c63facdf054cd64a0f42df94"
server_key_iv: "5f2b986b7894dfa696e698a4ddbf3085"
server_key_version: "encryptionkey@0"
I20260926 15:09:42.698606 20371 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260926 15:09:42.702124 20386 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:09:42.703423 20371 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 15:09:42.703692 20371 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435318278681-12427-0/minicluster-data/ts-2/data,/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435318278681-12427-0/minicluster-data/ts-2/wal
uuid: "3793e3bed5e04acb8adbc620baf660a4"
format_stamp: "Formatted at 2026-09-26 15:09:42 on dist-test-slave-865v"
server_key: "a880f1e4c63facdf054cd64a0f42df94"
server_key_iv: "5f2b986b7894dfa696e698a4ddbf3085"
server_key_version: "encryptionkey@0"
I20260926 15:09:42.703920 20371 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435318278681-12427-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435318278681-12427-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435318278681-12427-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:09:42.724617 20371 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:09:42.728912 20371 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:09:42.729240 20371 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:09:42.731868 20371 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:09:42.731967 20371 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:09:42.732105 20371 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:09:42.732189 20371 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 15:09:42 dist-test-slave-865v krb5kdc[19976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435382, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.34.195@KRBTEST.COM for oryx/127.12.34.254@KRBTEST.COM
I20260926 15:09:42.760581 20116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:42.732377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.195:53945 (local address 127.12.34.254:46481)
0926 15:09:42.732574 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:42.732583 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:42.734842 (+ 2259us) server_negotiation.cc:407] Connection header received
0926 15:09:42.737044 (+ 2202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:42.737049 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:42.737171 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:42.737298 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:09:42.739541 (+ 2243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:42.740598 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:42.743149 (+ 2551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:42.743712 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:42.743841 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:42.749648 (+ 5807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:09:42.749657 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:09:42.749666 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:09:42.749739 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:09:42.754920 (+ 5181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:42.756009 (+ 1089us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:42.756014 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:42.756019 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:42.756189 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:42.756754 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:42.756759 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:42.756763 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:42.757096 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:09:42.759538 (+ 2442us) server_negotiation.cc:1035] Waiting for connection context
0926 15:09:42.759780 (+ 242us) server_negotiation.cc:299] Negotiation successful
0926 15:09:42.760090 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 15:09:42.761166 20395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:42.732943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:46481 (local address 127.12.34.195:53945)
0926 15:09:42.734688 (+ 1745us) negotiation.cc:107] Waiting for socket to connect
0926 15:09:42.734734 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 15:09:42.736771 (+ 2037us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:09:42.737591 (+ 820us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:09:42.737607 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:09:42.738442 (+ 835us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:09:42.739341 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:42.739367 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:42.740791 (+ 1424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:42.740816 (+ 25us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:42.742988 (+ 2172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:42.743000 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:42.743952 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:42.743960 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:42.744285 (+ 325us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:42.745036 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:09:42.745076 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:09:42.749341 (+ 4265us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:09:42.755097 (+ 5756us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:09:42.755107 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:09:42.755125 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:09:42.755822 (+ 697us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:09:42.756362 (+ 540us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:09:42.756369 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:09:42.756374 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:09:42.756610 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:09:42.757254 (+ 644us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:09:42.757262 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:09:42.757767 (+ 505us) client_negotiation.cc:769] Sending connection context
0926 15:09:42.758097 (+ 330us) client_negotiation.cc:240] Negotiation successful
0926 15:09:42.758786 (+ 689us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1525,"thread_start_us":1332,"threads_started":1}
I20260926 15:09:42.778656 20228 heartbeater.cc:503] Master 127.12.34.254:46481 was elected leader, sending a full tablet report...
I20260926 15:09:42.787106 20116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:42.768774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.195:48803 (local address 127.12.34.254:46481)
0926 15:09:42.769238 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:42.769243 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:42.769362 (+ 119us) server_negotiation.cc:407] Connection header received
0926 15:09:42.769615 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:42.769624 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:42.769743 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:42.769868 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:09:42.771016 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:42.772443 (+ 1427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:42.774957 (+ 2514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:42.775872 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:42.776032 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:42.778680 (+ 2648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:09:42.778688 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:09:42.778695 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:09:42.778736 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:09:42.783330 (+ 4594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:42.784367 (+ 1037us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:42.784375 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:42.784381 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:42.784539 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:42.785029 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:42.785037 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:42.785042 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:42.785483 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:09:42.785632 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0926 15:09:42.786317 (+ 685us) server_negotiation.cc:299] Negotiation successful
0926 15:09:42.786613 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337}
I20260926 15:09:42.787122 20434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:42.768469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:46481 (local address 127.12.34.195:48803)
0926 15:09:42.769243 (+ 774us) negotiation.cc:107] Waiting for socket to connect
0926 15:09:42.769268 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 15:09:42.769427 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:09:42.769990 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:09:42.769998 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:09:42.770517 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:09:42.770861 (+ 344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:42.770873 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:42.772628 (+ 1755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:42.772636 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:42.774785 (+ 2149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:42.774803 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:42.776095 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:42.776103 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:42.776373 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:42.777181 (+ 808us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:09:42.777224 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:09:42.778442 (+ 1218us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:09:42.783508 (+ 5066us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:09:42.783516 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:09:42.783523 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:09:42.784206 (+ 683us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:09:42.784696 (+ 490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:09:42.784701 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:09:42.784704 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:09:42.784883 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:09:42.785718 (+ 835us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:09:42.785726 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:09:42.785966 (+ 240us) client_negotiation.cc:769] Sending connection context
0926 15:09:42.786124 (+ 158us) client_negotiation.cc:240] Negotiation successful
0926 15:09:42.786522 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"thread_start_us":364,"threads_started":1}
I20260926 15:09:42.812081 20371 rpc_server.cc:307] RPC server started. Bound to: 127.12.34.195:45155
I20260926 15:09:42.812105 20505 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.34.195:45155 every 8 connection(s)
I20260926 15:09:42.813409 20371 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435318278681-12427-0/minicluster-data/ts-2/data/info.pb
I20260926 15:09:42.815589 20506 heartbeater.cc:347] Connected to a master server at 127.12.34.254:46481
I20260926 15:09:42.815822 20506 heartbeater.cc:464] Registering TS with master...
I20260926 15:09:42.816622 20506 heartbeater.cc:511] Master 127.12.34.254:46481 requested a full tablet report, sending...
I20260926 15:09:42.818172 20021 ts_manager.cc:194] Registered new tserver with Master: 3793e3bed5e04acb8adbc620baf660a4 (127.12.34.195:45155)
I20260926 15:09:42.819514 20021 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.12.34.195@KRBTEST.COM'} at 127.12.34.195:53945
I20260926 15:09:42.820564 12427 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8uoJGv/build/asan/bin/kudu as pid 20371
I20260926 15:09:42.820737 12427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435318278681-12427-0/minicluster-data/ts-2/wal/instance
I20260926 15:09:42.821377 12427 external_mini_cluster.cc:1428] Setting key 82aadbceec1586f52f66fc602568f5be
I20260926 15:09:42.824301 12427 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 15:09:42 dist-test-slave-865v krb5kdc[19976](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435382, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 15:09:42 dist-test-slave-865v krb5kdc[19976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.12.34.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 15:09:42 dist-test-slave-865v krb5kdc[19976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.12.34.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 15:09:43.278738 20367 heartbeater.cc:503] Master 127.12.34.254:46481 was elected leader, sending a full tablet report...
I20260926 15:09:43.821782 20506 heartbeater.cc:503] Master 127.12.34.254:46481 was elected leader, sending a full tablet report...
W20260926 15:09:43.941339 20514 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.065s user 0.003s sys 0.012s
W20260926 15:09:43.942363 20116 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:09:42.868713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43056 (local address 127.12.34.254:46481)
0926 15:09:42.868929 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:42.868935 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:42.869290 (+ 355us) server_negotiation.cc:407] Connection header received
0926 15:09:42.869487 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:42.869491 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:42.869566 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:42.869651 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:09:42.871655 (+ 2004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:42.872376 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:42.874646 (+ 2270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:42.875041 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:42.875133 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:43.942149 (+1067016us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43056: BlockingRecv error: recv got EOF from 127.0.0.1:43056 (error 108)
Metrics: {"server-negotiator.queue_time_us":98}
Sep 26 15:09:43 dist-test-slave-865v krb5kdc[19976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435382, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.12.34.254@KRBTEST.COM
I20260926 15:09:43.965252 20116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:43.946941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43066 (local address 127.12.34.254:46481)
0926 15:09:43.947108 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:43.947113 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:43.947324 (+ 211us) server_negotiation.cc:407] Connection header received
0926 15:09:43.947440 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:43.947443 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:43.947557 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:43.947656 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:09:43.948738 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:43.949522 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:43.950792 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:43.951355 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:43.951659 (+ 304us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:43.960087 (+ 8428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:09:43.960092 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:09:43.960096 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:09:43.960128 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:09:43.962676 (+ 2548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:43.963319 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:43.963324 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:43.963326 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:43.963441 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:43.963838 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:43.963845 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:43.963848 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:43.964118 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:09:43.964414 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0926 15:09:43.964656 (+ 242us) server_negotiation.cc:299] Negotiation successful
0926 15:09:43.964915 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 15:09:43.969009 20021 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43066:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 15:09:43.975658 20021 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 15:09:43.997555 20527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:43.987056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.194:35581 (local address 127.0.0.1:41168)
0926 15:09:43.989106 (+ 2050us) negotiation.cc:107] Waiting for socket to connect
0926 15:09:43.989124 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 15:09:43.989226 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:09:43.990289 (+ 1063us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:09:43.990296 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:09:43.990330 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:09:43.991042 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:43.991052 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:43.992763 (+ 1711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:43.992769 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:43.995254 (+ 2485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:43.995268 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:43.996587 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:43.996595 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:43.996786 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:43.996853 (+ 67us) client_negotiation.cc:769] Sending connection context
0926 15:09:43.996952 (+ 99us) client_negotiation.cc:240] Negotiation successful
0926 15:09:43.997157 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1913,"mutex_wait_us":30,"thread_start_us":1594,"threads_started":1}
I20260926 15:09:43.998085 20526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:43.987010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37212 (local address 127.12.34.195:45155)
0926 15:09:43.989122 (+ 2112us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:43.989130 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:43.989160 (+ 30us) server_negotiation.cc:407] Connection header received
0926 15:09:43.989257 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:43.989266 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:43.989539 (+ 273us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:43.989724 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:09:43.991520 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:43.992893 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:43.995475 (+ 2582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:43.996849 (+ 1374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:43.996993 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:43.997227 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0926 15:09:43.997644 (+ 417us) server_negotiation.cc:299] Negotiation successful
0926 15:09:43.997801 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1933,"thread_start_us":2612,"threads_started":1}
I20260926 15:09:43.999112 20525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:43.986846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.195:45155 (local address 127.0.0.1:37212)
0926 15:09:43.988358 (+ 1512us) negotiation.cc:107] Waiting for socket to connect
0926 15:09:43.988398 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 15:09:43.988560 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:09:43.990456 (+ 1896us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:09:43.990464 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:09:43.990502 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:09:43.991303 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:43.991318 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:43.993030 (+ 1712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:43.993035 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:43.995318 (+ 2283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:43.995356 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:43.997049 (+ 1693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:43.997055 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:43.997226 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:43.997275 (+ 49us) client_negotiation.cc:769] Sending connection context
0926 15:09:43.997406 (+ 131us) client_negotiation.cc:240] Negotiation successful
0926 15:09:43.997534 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1263,"mutex_wait_us":4,"thread_start_us":1107,"threads_started":1}
I20260926 15:09:43.999248 20530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:43.989343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44032 (local address 127.12.34.193:44487)
0926 15:09:43.990293 (+ 950us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:43.990304 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:43.991737 (+ 1433us) server_negotiation.cc:407] Connection header received
0926 15:09:43.991868 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:43.991875 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:43.992069 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:43.992239 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:09:43.992782 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:43.994240 (+ 1458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:43.996663 (+ 2423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:43.998063 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:43.998194 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:43.998385 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0926 15:09:43.998893 (+ 508us) server_negotiation.cc:299] Negotiation successful
0926 15:09:43.999010 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":760,"thread_start_us":621,"threads_started":1}
I20260926 15:09:43.999410 20529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:43.988871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.193:44487 (local address 127.0.0.1:44032)
0926 15:09:43.991617 (+ 2746us) negotiation.cc:107] Waiting for socket to connect
0926 15:09:43.991635 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 15:09:43.991744 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:09:43.992249 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:09:43.992253 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:09:43.992276 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:09:43.992583 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:43.992596 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:43.994378 (+ 1782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:43.994383 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:43.996507 (+ 2124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:43.996522 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:43.998507 (+ 1985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:43.998513 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:43.998722 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:43.998754 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 15:09:43.998860 (+ 106us) client_negotiation.cc:240] Negotiation successful
0926 15:09:43.998982 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2611,"mutex_wait_us":24,"thread_start_us":2459,"threads_started":1}
I20260926 15:09:44.000181 20528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:43.987701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41168 (local address 127.12.34.194:35581)
0926 15:09:43.989695 (+ 1994us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:43.989703 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:43.989727 (+ 24us) server_negotiation.cc:407] Connection header received
0926 15:09:43.989854 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:43.989861 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:43.990073 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:43.990517 (+ 444us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:09:43.991227 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:43.992654 (+ 1427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:43.995415 (+ 2761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:43.996321 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:43.998376 (+ 2055us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:43.998554 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0926 15:09:43.998614 (+ 60us) server_negotiation.cc:299] Negotiation successful
0926 15:09:43.999885 (+ 1271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1831,"thread_start_us":1703,"threads_started":1}
I20260926 15:09:44.002768 20441 tablet_service.cc:1467] Processing CreateTablet for tablet f2b9869c29e24ecba83b6bf5acdc055e (DEFAULT_TABLE table=test-table [id=6ea64069fb884ba79adedad5afcba4c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:09:44.002866 20158 tablet_service.cc:1467] Processing CreateTablet for tablet f2b9869c29e24ecba83b6bf5acdc055e (DEFAULT_TABLE table=test-table [id=6ea64069fb884ba79adedad5afcba4c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:09:44.003433 20297 tablet_service.cc:1467] Processing CreateTablet for tablet f2b9869c29e24ecba83b6bf5acdc055e (DEFAULT_TABLE table=test-table [id=6ea64069fb884ba79adedad5afcba4c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:09:44.004004 20441 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2b9869c29e24ecba83b6bf5acdc055e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:09:44.004163 20158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2b9869c29e24ecba83b6bf5acdc055e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:09:44.004586 20297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2b9869c29e24ecba83b6bf5acdc055e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:09:44.016774 20531 tablet_bootstrap.cc:492] T f2b9869c29e24ecba83b6bf5acdc055e P 3793e3bed5e04acb8adbc620baf660a4: Bootstrap starting.
I20260926 15:09:44.020153 20531 tablet_bootstrap.cc:654] T f2b9869c29e24ecba83b6bf5acdc055e P 3793e3bed5e04acb8adbc620baf660a4: Neither blocks nor log segments found. Creating new log.
I20260926 15:09:44.021332 20531 log.cc:824] T f2b9869c29e24ecba83b6bf5acdc055e P 3793e3bed5e04acb8adbc620baf660a4: Log is configured to *not* fsync() on all Append() calls
I20260926 15:09:44.022182 20533 tablet_bootstrap.cc:492] T f2b9869c29e24ecba83b6bf5acdc055e P bffb811d93894ea5a14e6f4f56f4a196: Bootstrap starting.
I20260926 15:09:44.022719 20532 tablet_bootstrap.cc:492] T f2b9869c29e24ecba83b6bf5acdc055e P 641358d46138484794ffdf3e0ceb4109: Bootstrap starting.
I20260926 15:09:44.024718 20531 tablet_bootstrap.cc:492] T f2b9869c29e24ecba83b6bf5acdc055e P 3793e3bed5e04acb8adbc620baf660a4: No bootstrap required, opened a new log
I20260926 15:09:44.024986 20531 ts_tablet_manager.cc:1397] T f2b9869c29e24ecba83b6bf5acdc055e P 3793e3bed5e04acb8adbc620baf660a4: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260926 15:09:44.026166 20533 tablet_bootstrap.cc:654] T f2b9869c29e24ecba83b6bf5acdc055e P bffb811d93894ea5a14e6f4f56f4a196: Neither blocks nor log segments found. Creating new log.
I20260926 15:09:44.026165 20532 tablet_bootstrap.cc:654] T f2b9869c29e24ecba83b6bf5acdc055e P 641358d46138484794ffdf3e0ceb4109: Neither blocks nor log segments found. Creating new log.
I20260926 15:09:44.027371 20533 log.cc:824] T f2b9869c29e24ecba83b6bf5acdc055e P bffb811d93894ea5a14e6f4f56f4a196: Log is configured to *not* fsync() on all Append() calls
I20260926 15:09:44.027371 20532 log.cc:824] T f2b9869c29e24ecba83b6bf5acdc055e P 641358d46138484794ffdf3e0ceb4109: Log is configured to *not* fsync() on all Append() calls
I20260926 15:09:44.031616 20533 tablet_bootstrap.cc:492] T f2b9869c29e24ecba83b6bf5acdc055e P bffb811d93894ea5a14e6f4f56f4a196: No bootstrap required, opened a new log
I20260926 15:09:44.031724 20532 tablet_bootstrap.cc:492] T f2b9869c29e24ecba83b6bf5acdc055e P 641358d46138484794ffdf3e0ceb4109: No bootstrap required, opened a new log
I20260926 15:09:44.031873 20533 ts_tablet_manager.cc:1397] T f2b9869c29e24ecba83b6bf5acdc055e P bffb811d93894ea5a14e6f4f56f4a196: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.004s
I20260926 15:09:44.031949 20532 ts_tablet_manager.cc:1397] T f2b9869c29e24ecba83b6bf5acdc055e P 641358d46138484794ffdf3e0ceb4109: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260926 15:09:44.032934 20531 raft_consensus.cc:348] T f2b9869c29e24ecba83b6bf5acdc055e P 3793e3bed5e04acb8adbc620baf660a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3793e3bed5e04acb8adbc620baf660a4" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 45155 } } peers { permanent_uuid: "641358d46138484794ffdf3e0ceb4109" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 35581 } } peers { permanent_uuid: "bffb811d93894ea5a14e6f4f56f4a196" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 44487 } }
I20260926 15:09:44.033349 20531 raft_consensus.cc:374] T f2b9869c29e24ecba83b6bf5acdc055e P 3793e3bed5e04acb8adbc620baf660a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:09:44.033440 20531 raft_consensus.cc:729] T f2b9869c29e24ecba83b6bf5acdc055e P 3793e3bed5e04acb8adbc620baf660a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3793e3bed5e04acb8adbc620baf660a4, State: Initialized, Role: FOLLOWER
I20260926 15:09:44.034003 20531 consensus_queue.cc:260] T f2b9869c29e24ecba83b6bf5acdc055e P 3793e3bed5e04acb8adbc620baf660a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3793e3bed5e04acb8adbc620baf660a4" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 45155 } } peers { permanent_uuid: "641358d46138484794ffdf3e0ceb4109" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 35581 } } peers { permanent_uuid: "bffb811d93894ea5a14e6f4f56f4a196" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 44487 } }
I20260926 15:09:44.036589 20531 ts_tablet_manager.cc:1428] T f2b9869c29e24ecba83b6bf5acdc055e P 3793e3bed5e04acb8adbc620baf660a4: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260926 15:09:44.042508 20533 raft_consensus.cc:348] T f2b9869c29e24ecba83b6bf5acdc055e P bffb811d93894ea5a14e6f4f56f4a196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3793e3bed5e04acb8adbc620baf660a4" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 45155 } } peers { permanent_uuid: "641358d46138484794ffdf3e0ceb4109" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 35581 } } peers { permanent_uuid: "bffb811d93894ea5a14e6f4f56f4a196" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 44487 } }
I20260926 15:09:44.042508 20532 raft_consensus.cc:348] T f2b9869c29e24ecba83b6bf5acdc055e P 641358d46138484794ffdf3e0ceb4109 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3793e3bed5e04acb8adbc620baf660a4" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 45155 } } peers { permanent_uuid: "641358d46138484794ffdf3e0ceb4109" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 35581 } } peers { permanent_uuid: "bffb811d93894ea5a14e6f4f56f4a196" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 44487 } }
I20260926 15:09:44.042917 20533 raft_consensus.cc:374] T f2b9869c29e24ecba83b6bf5acdc055e P bffb811d93894ea5a14e6f4f56f4a196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:09:44.042917 20532 raft_consensus.cc:374] T f2b9869c29e24ecba83b6bf5acdc055e P 641358d46138484794ffdf3e0ceb4109 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:09:44.043017 20533 raft_consensus.cc:729] T f2b9869c29e24ecba83b6bf5acdc055e P bffb811d93894ea5a14e6f4f56f4a196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bffb811d93894ea5a14e6f4f56f4a196, State: Initialized, Role: FOLLOWER
I20260926 15:09:44.043017 20532 raft_consensus.cc:729] T f2b9869c29e24ecba83b6bf5acdc055e P 641358d46138484794ffdf3e0ceb4109 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 641358d46138484794ffdf3e0ceb4109, State: Initialized, Role: FOLLOWER
I20260926 15:09:44.043594 20533 consensus_queue.cc:260] T f2b9869c29e24ecba83b6bf5acdc055e P bffb811d93894ea5a14e6f4f56f4a196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3793e3bed5e04acb8adbc620baf660a4" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 45155 } } peers { permanent_uuid: "641358d46138484794ffdf3e0ceb4109" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 35581 } } peers { permanent_uuid: "bffb811d93894ea5a14e6f4f56f4a196" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 44487 } }
I20260926 15:09:44.043594 20532 consensus_queue.cc:260] T f2b9869c29e24ecba83b6bf5acdc055e P 641358d46138484794ffdf3e0ceb4109 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3793e3bed5e04acb8adbc620baf660a4" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 45155 } } peers { permanent_uuid: "641358d46138484794ffdf3e0ceb4109" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 35581 } } peers { permanent_uuid: "bffb811d93894ea5a14e6f4f56f4a196" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 44487 } }
I20260926 15:09:44.045439 20533 ts_tablet_manager.cc:1428] T f2b9869c29e24ecba83b6bf5acdc055e P bffb811d93894ea5a14e6f4f56f4a196: Time spent starting tablet: real 0.013s user 0.015s sys 0.001s
I20260926 15:09:44.045724 20532 ts_tablet_manager.cc:1428] T f2b9869c29e24ecba83b6bf5acdc055e P 641358d46138484794ffdf3e0ceb4109: Time spent starting tablet: real 0.014s user 0.010s sys 0.005s
W20260926 15:09:44.065872 20507 tablet.cc:2369] T f2b9869c29e24ecba83b6bf5acdc055e P 3793e3bed5e04acb8adbc620baf660a4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 15:09:44.271322 20229 tablet.cc:2369] T f2b9869c29e24ecba83b6bf5acdc055e P bffb811d93894ea5a14e6f4f56f4a196: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 15:09:44.274892 20368 tablet.cc:2369] T f2b9869c29e24ecba83b6bf5acdc055e P 641358d46138484794ffdf3e0ceb4109: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 15:09:44.281915 20537 raft_consensus.cc:482] T f2b9869c29e24ecba83b6bf5acdc055e P 3793e3bed5e04acb8adbc620baf660a4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 15:09:44.282217 20537 raft_consensus.cc:504] T f2b9869c29e24ecba83b6bf5acdc055e P 3793e3bed5e04acb8adbc620baf660a4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3793e3bed5e04acb8adbc620baf660a4" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 45155 } } peers { permanent_uuid: "641358d46138484794ffdf3e0ceb4109" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 35581 } } peers { permanent_uuid: "bffb811d93894ea5a14e6f4f56f4a196" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 44487 } }
I20260926 15:09:44.283821 20537 leader_election.cc:288] T f2b9869c29e24ecba83b6bf5acdc055e P 3793e3bed5e04acb8adbc620baf660a4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 641358d46138484794ffdf3e0ceb4109 (127.12.34.194:35581), bffb811d93894ea5a14e6f4f56f4a196 (127.12.34.193:44487)
I20260926 15:09:44.290383 20530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:44.284344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.195:60459 (local address 127.12.34.193:44487)
0926 15:09:44.284576 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:44.284585 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:44.285005 (+ 420us) server_negotiation.cc:407] Connection header received
0926 15:09:44.285259 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:44.285263 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:44.285318 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:44.285426 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:09:44.285962 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:44.286663 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:44.288545 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:44.289482 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:44.289654 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:44.289732 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 15:09:44.290021 (+ 289us) server_negotiation.cc:299] Negotiation successful
0926 15:09:44.290139 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260926 15:09:44.290426 20528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:44.284188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.195:41305 (local address 127.12.34.194:35581)
0926 15:09:44.284411 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:44.284418 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:44.285136 (+ 718us) server_negotiation.cc:407] Connection header received
0926 15:09:44.285339 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:44.285343 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:44.285394 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:44.285490 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:09:44.286086 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:44.286753 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:44.288556 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:44.289481 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:44.289683 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:44.289761 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 15:09:44.290057 (+ 296us) server_negotiation.cc:299] Negotiation successful
0926 15:09:44.290174 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 15:09:44.290542 20540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:44.284156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.193:44487 (local address 127.12.34.195:60459)
0926 15:09:44.284900 (+ 744us) negotiation.cc:107] Waiting for socket to connect
0926 15:09:44.284915 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 15:09:44.285118 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:09:44.285442 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:09:44.285446 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:09:44.285470 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:09:44.285807 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:44.285815 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:44.286795 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:44.286803 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:44.288398 (+ 1595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:44.288409 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:44.289698 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:44.289704 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:44.289844 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:44.289886 (+ 42us) client_negotiation.cc:769] Sending connection context
0926 15:09:44.289997 (+ 111us) client_negotiation.cc:240] Negotiation successful
0926 15:09:44.290118 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":590,"mutex_wait_us":45,"thread_start_us":370,"threads_started":1}
I20260926 15:09:44.290938 20541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:44.284192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.194:35581 (local address 127.12.34.195:41305)
0926 15:09:44.285025 (+ 833us) negotiation.cc:107] Waiting for socket to connect
0926 15:09:44.285039 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 15:09:44.285202 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:09:44.285519 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:09:44.285524 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:09:44.285560 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:09:44.285948 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:44.285956 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:44.286962 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:44.286968 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:44.288408 (+ 1440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:44.288419 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:44.289744 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:44.289750 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:44.289910 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:44.289932 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 15:09:44.290033 (+ 101us) client_negotiation.cc:240] Negotiation successful
0926 15:09:44.290152 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":715,"mutex_wait_us":109,"thread_start_us":418,"threads_started":1}
I20260926 15:09:44.291572 20178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2b9869c29e24ecba83b6bf5acdc055e" candidate_uuid: "3793e3bed5e04acb8adbc620baf660a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bffb811d93894ea5a14e6f4f56f4a196" is_pre_election: true
I20260926 15:09:44.291864 20322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2b9869c29e24ecba83b6bf5acdc055e" candidate_uuid: "3793e3bed5e04acb8adbc620baf660a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "641358d46138484794ffdf3e0ceb4109" is_pre_election: true
I20260926 15:09:44.292042 20178 raft_consensus.cc:2454] T f2b9869c29e24ecba83b6bf5acdc055e P bffb811d93894ea5a14e6f4f56f4a196 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3793e3bed5e04acb8adbc620baf660a4 in term 0.
I20260926 15:09:44.292317 20322 raft_consensus.cc:2454] T f2b9869c29e24ecba83b6bf5acdc055e P 641358d46138484794ffdf3e0ceb4109 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3793e3bed5e04acb8adbc620baf660a4 in term 0.
I20260926 15:09:44.292723 20390 leader_election.cc:302] T f2b9869c29e24ecba83b6bf5acdc055e P 3793e3bed5e04acb8adbc620baf660a4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3793e3bed5e04acb8adbc620baf660a4, bffb811d93894ea5a14e6f4f56f4a196; no voters:
I20260926 15:09:44.293125 20537 raft_consensus.cc:2781] T f2b9869c29e24ecba83b6bf5acdc055e P 3793e3bed5e04acb8adbc620baf660a4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 15:09:44.293282 20537 raft_consensus.cc:482] T f2b9869c29e24ecba83b6bf5acdc055e P 3793e3bed5e04acb8adbc620baf660a4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 15:09:44.293381 20537 raft_consensus.cc:3037] T f2b9869c29e24ecba83b6bf5acdc055e P 3793e3bed5e04acb8adbc620baf660a4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:09:44.295464 20537 raft_consensus.cc:504] T f2b9869c29e24ecba83b6bf5acdc055e P 3793e3bed5e04acb8adbc620baf660a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3793e3bed5e04acb8adbc620baf660a4" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 45155 } } peers { permanent_uuid: "641358d46138484794ffdf3e0ceb4109" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 35581 } } peers { permanent_uuid: "bffb811d93894ea5a14e6f4f56f4a196" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 44487 } }
I20260926 15:09:44.296084 20537 leader_election.cc:288] T f2b9869c29e24ecba83b6bf5acdc055e P 3793e3bed5e04acb8adbc620baf660a4 [CANDIDATE]: Term 1 election: Requested vote from peers 641358d46138484794ffdf3e0ceb4109 (127.12.34.194:35581), bffb811d93894ea5a14e6f4f56f4a196 (127.12.34.193:44487)
I20260926 15:09:44.296630 20322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2b9869c29e24ecba83b6bf5acdc055e" candidate_uuid: "3793e3bed5e04acb8adbc620baf660a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "641358d46138484794ffdf3e0ceb4109"
I20260926 15:09:44.296828 20322 raft_consensus.cc:3037] T f2b9869c29e24ecba83b6bf5acdc055e P 641358d46138484794ffdf3e0ceb4109 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:09:44.297348 20178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2b9869c29e24ecba83b6bf5acdc055e" candidate_uuid: "3793e3bed5e04acb8adbc620baf660a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bffb811d93894ea5a14e6f4f56f4a196"
I20260926 15:09:44.297508 20178 raft_consensus.cc:3037] T f2b9869c29e24ecba83b6bf5acdc055e P bffb811d93894ea5a14e6f4f56f4a196 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:09:44.299271 20322 raft_consensus.cc:2454] T f2b9869c29e24ecba83b6bf5acdc055e P 641358d46138484794ffdf3e0ceb4109 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3793e3bed5e04acb8adbc620baf660a4 in term 1.
I20260926 15:09:44.299636 20388 leader_election.cc:302] T f2b9869c29e24ecba83b6bf5acdc055e P 3793e3bed5e04acb8adbc620baf660a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3793e3bed5e04acb8adbc620baf660a4, 641358d46138484794ffdf3e0ceb4109; no voters:
I20260926 15:09:44.299930 20537 raft_consensus.cc:2781] T f2b9869c29e24ecba83b6bf5acdc055e P 3793e3bed5e04acb8adbc620baf660a4 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:09:44.300063 20178 raft_consensus.cc:2454] T f2b9869c29e24ecba83b6bf5acdc055e P bffb811d93894ea5a14e6f4f56f4a196 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3793e3bed5e04acb8adbc620baf660a4 in term 1.
I20260926 15:09:44.300534 20537 raft_consensus.cc:686] T f2b9869c29e24ecba83b6bf5acdc055e P 3793e3bed5e04acb8adbc620baf660a4 [term 1 LEADER]: Becoming Leader. State: Replica: 3793e3bed5e04acb8adbc620baf660a4, State: Running, Role: LEADER
I20260926 15:09:44.301340 20537 consensus_queue.cc:237] T f2b9869c29e24ecba83b6bf5acdc055e P 3793e3bed5e04acb8adbc620baf660a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3793e3bed5e04acb8adbc620baf660a4" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 45155 } } peers { permanent_uuid: "641358d46138484794ffdf3e0ceb4109" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 35581 } } peers { permanent_uuid: "bffb811d93894ea5a14e6f4f56f4a196" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 44487 } }
I20260926 15:09:44.305567 20021 catalog_manager.cc:5495] T f2b9869c29e24ecba83b6bf5acdc055e P 3793e3bed5e04acb8adbc620baf660a4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3793e3bed5e04acb8adbc620baf660a4 (127.12.34.195). New cstate: current_term: 1 leader_uuid: "3793e3bed5e04acb8adbc620baf660a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3793e3bed5e04acb8adbc620baf660a4" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 45155 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "641358d46138484794ffdf3e0ceb4109" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 35581 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bffb811d93894ea5a14e6f4f56f4a196" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 44487 } health_report { overall_health: UNKNOWN } } }
I20260926 15:09:44.364271 20526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:44.359268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37224 (local address 127.12.34.195:45155)
0926 15:09:44.359530 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:44.359535 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:44.359552 (+ 17us) server_negotiation.cc:407] Connection header received
0926 15:09:44.359646 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:44.359650 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:44.359721 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:44.359885 (+ 164us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 15:09:44.360211 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:44.360896 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:44.362227 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:44.362624 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:44.362760 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:44.363189 (+ 429us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 15:09:44.363693 (+ 504us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 15:09:44.363843 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0926 15:09:44.363938 (+ 95us) server_negotiation.cc:299] Negotiation successful
0926 15:09:44.364015 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260926 15:09:44.371973 20178 raft_consensus.cc:1261] T f2b9869c29e24ecba83b6bf5acdc055e P bffb811d93894ea5a14e6f4f56f4a196 [term 1 FOLLOWER]: Refusing update from remote peer 3793e3bed5e04acb8adbc620baf660a4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:09:44.371973 20322 raft_consensus.cc:1261] T f2b9869c29e24ecba83b6bf5acdc055e P 641358d46138484794ffdf3e0ceb4109 [term 1 FOLLOWER]: Refusing update from remote peer 3793e3bed5e04acb8adbc620baf660a4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:09:44.372918 20537 consensus_queue.cc:1035] T f2b9869c29e24ecba83b6bf5acdc055e P 3793e3bed5e04acb8adbc620baf660a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bffb811d93894ea5a14e6f4f56f4a196" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 44487 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:09:44.373234 20542 consensus_queue.cc:1035] T f2b9869c29e24ecba83b6bf5acdc055e P 3793e3bed5e04acb8adbc620baf660a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "641358d46138484794ffdf3e0ceb4109" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 35581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:09:44.384315 20545 mvcc.cc:205] Tried to move back new op lower bound from 7333623334378938368 to 7333623334101684224. Current Snapshot: MvccSnapshot[applied={T|T < 7333623334378938368}]
I20260926 15:09:44.384632 20546 mvcc.cc:205] Tried to move back new op lower bound from 7333623334378938368 to 7333623334101684224. Current Snapshot: MvccSnapshot[applied={T|T < 7333623334378938368}]
I20260926 15:09:44.387377 20547 mvcc.cc:205] Tried to move back new op lower bound from 7333623334378938368 to 7333623334101684224. Current Snapshot: MvccSnapshot[applied={T|T < 7333623334378938368}]
I20260926 15:09:44.403218 20020 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43066:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 15:09:44.403519 20020 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43066:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 15:09:44.407234 20020 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1a3d30fc883a4f67bc16c22629712a96: Sending DeleteTablet for 3 replicas of tablet f2b9869c29e24ecba83b6bf5acdc055e
I20260926 15:09:44.408367 20441 tablet_service.cc:1514] Processing DeleteTablet for tablet f2b9869c29e24ecba83b6bf5acdc055e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:09:44 UTC) from {username='oryx', principal='oryx/127.12.34.254@KRBTEST.COM'} at 127.0.0.1:37212
I20260926 15:09:44.408517 20297 tablet_service.cc:1514] Processing DeleteTablet for tablet f2b9869c29e24ecba83b6bf5acdc055e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:09:44 UTC) from {username='oryx', principal='oryx/127.12.34.254@KRBTEST.COM'} at 127.0.0.1:41168
I20260926 15:09:44.408979 20158 tablet_service.cc:1514] Processing DeleteTablet for tablet f2b9869c29e24ecba83b6bf5acdc055e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:09:44 UTC) from {username='oryx', principal='oryx/127.12.34.254@KRBTEST.COM'} at 127.0.0.1:44032
I20260926 15:09:44.410085 20558 tablet_replica.cc:330] T f2b9869c29e24ecba83b6bf5acdc055e P bffb811d93894ea5a14e6f4f56f4a196: stopping tablet replica
I20260926 15:09:44.410159 20020 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43066:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 15:09:44.411033 20020 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 15:09:44.411011 20558 raft_consensus.cc:2229] T f2b9869c29e24ecba83b6bf5acdc055e P bffb811d93894ea5a14e6f4f56f4a196 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 15:09:44.411851 20558 raft_consensus.cc:2258] T f2b9869c29e24ecba83b6bf5acdc055e P bffb811d93894ea5a14e6f4f56f4a196 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:09:44.413309 20557 tablet_replica.cc:330] T f2b9869c29e24ecba83b6bf5acdc055e P 641358d46138484794ffdf3e0ceb4109: stopping tablet replica
I20260926 15:09:44.413878 20557 raft_consensus.cc:2229] T f2b9869c29e24ecba83b6bf5acdc055e P 641358d46138484794ffdf3e0ceb4109 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 15:09:44.414376 20558 ts_tablet_manager.cc:1905] T f2b9869c29e24ecba83b6bf5acdc055e P bffb811d93894ea5a14e6f4f56f4a196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:09:44.414382 20557 raft_consensus.cc:2258] T f2b9869c29e24ecba83b6bf5acdc055e P 641358d46138484794ffdf3e0ceb4109 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:09:44.421792 20158 tablet_service.cc:1467] Processing CreateTablet for tablet dc00a892c09141d8ab443a99a16db082 (DEFAULT_TABLE table=test-table [id=fc77104f72c64155a07f00405d7fed6e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:09:44.422215 20296 tablet_service.cc:1467] Processing CreateTablet for tablet dc00a892c09141d8ab443a99a16db082 (DEFAULT_TABLE table=test-table [id=fc77104f72c64155a07f00405d7fed6e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:09:44.422680 20296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc00a892c09141d8ab443a99a16db082. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:09:44.422569 20440 tablet_service.cc:1467] Processing CreateTablet for tablet dc00a892c09141d8ab443a99a16db082 (DEFAULT_TABLE table=test-table [id=fc77104f72c64155a07f00405d7fed6e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:09:44.423152 20440 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc00a892c09141d8ab443a99a16db082. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:09:44.426373 20158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc00a892c09141d8ab443a99a16db082. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:09:44.426431 20558 ts_tablet_manager.cc:1918] T f2b9869c29e24ecba83b6bf5acdc055e P bffb811d93894ea5a14e6f4f56f4a196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 15:09:44.426674 20558 log.cc:1196] T f2b9869c29e24ecba83b6bf5acdc055e P bffb811d93894ea5a14e6f4f56f4a196: Deleting WAL directory at /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435318278681-12427-0/minicluster-data/ts-0/wal/wals/f2b9869c29e24ecba83b6bf5acdc055e
I20260926 15:09:44.427337 20558 ts_tablet_manager.cc:1939] T f2b9869c29e24ecba83b6bf5acdc055e P bffb811d93894ea5a14e6f4f56f4a196: Deleting consensus metadata
I20260926 15:09:44.428869 20532 tablet_bootstrap.cc:492] T dc00a892c09141d8ab443a99a16db082 P 641358d46138484794ffdf3e0ceb4109: Bootstrap starting.
I20260926 15:09:44.430116 20008 catalog_manager.cc:4841] TS bffb811d93894ea5a14e6f4f56f4a196 (127.12.34.193:44487): tablet f2b9869c29e24ecba83b6bf5acdc055e (table test-table [id=6ea64069fb884ba79adedad5afcba4c6]) successfully deleted
I20260926 15:09:44.430449 20532 tablet_bootstrap.cc:654] T dc00a892c09141d8ab443a99a16db082 P 641358d46138484794ffdf3e0ceb4109: Neither blocks nor log segments found. Creating new log.
I20260926 15:09:44.430371 20557 ts_tablet_manager.cc:1905] T f2b9869c29e24ecba83b6bf5acdc055e P 641358d46138484794ffdf3e0ceb4109: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:09:44.434787 20532 tablet_bootstrap.cc:492] T dc00a892c09141d8ab443a99a16db082 P 641358d46138484794ffdf3e0ceb4109: No bootstrap required, opened a new log
I20260926 15:09:44.435022 20532 ts_tablet_manager.cc:1397] T dc00a892c09141d8ab443a99a16db082 P 641358d46138484794ffdf3e0ceb4109: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260926 15:09:44.436065 20532 raft_consensus.cc:348] T dc00a892c09141d8ab443a99a16db082 P 641358d46138484794ffdf3e0ceb4109 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bffb811d93894ea5a14e6f4f56f4a196" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 44487 } } peers { permanent_uuid: "641358d46138484794ffdf3e0ceb4109" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 35581 } } peers { permanent_uuid: "3793e3bed5e04acb8adbc620baf660a4" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 45155 } }
I20260926 15:09:44.436396 20532 raft_consensus.cc:374] T dc00a892c09141d8ab443a99a16db082 P 641358d46138484794ffdf3e0ceb4109 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:09:44.436553 20532 raft_consensus.cc:729] T dc00a892c09141d8ab443a99a16db082 P 641358d46138484794ffdf3e0ceb4109 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 641358d46138484794ffdf3e0ceb4109, State: Initialized, Role: FOLLOWER
I20260926 15:09:44.437285 20531 tablet_bootstrap.cc:492] T dc00a892c09141d8ab443a99a16db082 P 3793e3bed5e04acb8adbc620baf660a4: Bootstrap starting.
I20260926 15:09:44.437778 20533 tablet_bootstrap.cc:492] T dc00a892c09141d8ab443a99a16db082 P bffb811d93894ea5a14e6f4f56f4a196: Bootstrap starting.
I20260926 15:09:44.437829 20556 tablet_replica.cc:330] T f2b9869c29e24ecba83b6bf5acdc055e P 3793e3bed5e04acb8adbc620baf660a4: stopping tablet replica
I20260926 15:09:44.438086 20557 ts_tablet_manager.cc:1918] T f2b9869c29e24ecba83b6bf5acdc055e P 641358d46138484794ffdf3e0ceb4109: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 15:09:44.438213 20557 log.cc:1196] T f2b9869c29e24ecba83b6bf5acdc055e P 641358d46138484794ffdf3e0ceb4109: Deleting WAL directory at /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435318278681-12427-0/minicluster-data/ts-1/wal/wals/f2b9869c29e24ecba83b6bf5acdc055e
I20260926 15:09:44.438297 20556 raft_consensus.cc:2229] T f2b9869c29e24ecba83b6bf5acdc055e P 3793e3bed5e04acb8adbc620baf660a4 [term 1 LEADER]: Raft consensus shutting down.
I20260926 15:09:44.438697 20557 ts_tablet_manager.cc:1939] T f2b9869c29e24ecba83b6bf5acdc055e P 641358d46138484794ffdf3e0ceb4109: Deleting consensus metadata
I20260926 15:09:44.439065 20531 tablet_bootstrap.cc:654] T dc00a892c09141d8ab443a99a16db082 P 3793e3bed5e04acb8adbc620baf660a4: Neither blocks nor log segments found. Creating new log.
I20260926 15:09:44.439353 20533 tablet_bootstrap.cc:654] T dc00a892c09141d8ab443a99a16db082 P bffb811d93894ea5a14e6f4f56f4a196: Neither blocks nor log segments found. Creating new log.
I20260926 15:09:44.436946 20532 consensus_queue.cc:260] T dc00a892c09141d8ab443a99a16db082 P 641358d46138484794ffdf3e0ceb4109 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bffb811d93894ea5a14e6f4f56f4a196" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 44487 } } peers { permanent_uuid: "641358d46138484794ffdf3e0ceb4109" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 35581 } } peers { permanent_uuid: "3793e3bed5e04acb8adbc620baf660a4" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 45155 } }
I20260926 15:09:44.440239 20532 ts_tablet_manager.cc:1428] T dc00a892c09141d8ab443a99a16db082 P 641358d46138484794ffdf3e0ceb4109: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260926 15:09:44.440325 20006 catalog_manager.cc:4841] TS 641358d46138484794ffdf3e0ceb4109 (127.12.34.194:35581): tablet f2b9869c29e24ecba83b6bf5acdc055e (table test-table [id=6ea64069fb884ba79adedad5afcba4c6]) successfully deleted
I20260926 15:09:44.441828 20533 tablet_bootstrap.cc:492] T dc00a892c09141d8ab443a99a16db082 P bffb811d93894ea5a14e6f4f56f4a196: No bootstrap required, opened a new log
I20260926 15:09:44.441972 20533 ts_tablet_manager.cc:1397] T dc00a892c09141d8ab443a99a16db082 P bffb811d93894ea5a14e6f4f56f4a196: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260926 15:09:44.442750 20533 raft_consensus.cc:348] T dc00a892c09141d8ab443a99a16db082 P bffb811d93894ea5a14e6f4f56f4a196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bffb811d93894ea5a14e6f4f56f4a196" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 44487 } } peers { permanent_uuid: "641358d46138484794ffdf3e0ceb4109" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 35581 } } peers { permanent_uuid: "3793e3bed5e04acb8adbc620baf660a4" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 45155 } }
I20260926 15:09:44.442966 20533 raft_consensus.cc:374] T dc00a892c09141d8ab443a99a16db082 P bffb811d93894ea5a14e6f4f56f4a196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:09:44.443061 20533 raft_consensus.cc:729] T dc00a892c09141d8ab443a99a16db082 P bffb811d93894ea5a14e6f4f56f4a196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bffb811d93894ea5a14e6f4f56f4a196, State: Initialized, Role: FOLLOWER
I20260926 15:09:44.443306 20533 consensus_queue.cc:260] T dc00a892c09141d8ab443a99a16db082 P bffb811d93894ea5a14e6f4f56f4a196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bffb811d93894ea5a14e6f4f56f4a196" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 44487 } } peers { permanent_uuid: "641358d46138484794ffdf3e0ceb4109" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 35581 } } peers { permanent_uuid: "3793e3bed5e04acb8adbc620baf660a4" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 45155 } }
I20260926 15:09:44.443876 20533 ts_tablet_manager.cc:1428] T dc00a892c09141d8ab443a99a16db082 P bffb811d93894ea5a14e6f4f56f4a196: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260926 15:09:44.439083 20556 raft_consensus.cc:2258] T f2b9869c29e24ecba83b6bf5acdc055e P 3793e3bed5e04acb8adbc620baf660a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:09:44.446134 20556 ts_tablet_manager.cc:1905] T f2b9869c29e24ecba83b6bf5acdc055e P 3793e3bed5e04acb8adbc620baf660a4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:09:44.453161 20556 ts_tablet_manager.cc:1918] T f2b9869c29e24ecba83b6bf5acdc055e P 3793e3bed5e04acb8adbc620baf660a4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 15:09:44.453331 20556 log.cc:1196] T f2b9869c29e24ecba83b6bf5acdc055e P 3793e3bed5e04acb8adbc620baf660a4: Deleting WAL directory at /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435318278681-12427-0/minicluster-data/ts-2/wal/wals/f2b9869c29e24ecba83b6bf5acdc055e
I20260926 15:09:44.453961 20556 ts_tablet_manager.cc:1939] T f2b9869c29e24ecba83b6bf5acdc055e P 3793e3bed5e04acb8adbc620baf660a4: Deleting consensus metadata
I20260926 15:09:44.454591 20531 tablet_bootstrap.cc:492] T dc00a892c09141d8ab443a99a16db082 P 3793e3bed5e04acb8adbc620baf660a4: No bootstrap required, opened a new log
I20260926 15:09:44.454764 20531 ts_tablet_manager.cc:1397] T dc00a892c09141d8ab443a99a16db082 P 3793e3bed5e04acb8adbc620baf660a4: Time spent bootstrapping tablet: real 0.018s user 0.006s sys 0.009s
I20260926 15:09:44.455542 20005 catalog_manager.cc:4841] TS 3793e3bed5e04acb8adbc620baf660a4 (127.12.34.195:45155): tablet f2b9869c29e24ecba83b6bf5acdc055e (table test-table [id=6ea64069fb884ba79adedad5afcba4c6]) successfully deleted
I20260926 15:09:44.455734 20531 raft_consensus.cc:348] T dc00a892c09141d8ab443a99a16db082 P 3793e3bed5e04acb8adbc620baf660a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bffb811d93894ea5a14e6f4f56f4a196" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 44487 } } peers { permanent_uuid: "641358d46138484794ffdf3e0ceb4109" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 35581 } } peers { permanent_uuid: "3793e3bed5e04acb8adbc620baf660a4" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 45155 } }
I20260926 15:09:44.455977 20531 raft_consensus.cc:374] T dc00a892c09141d8ab443a99a16db082 P 3793e3bed5e04acb8adbc620baf660a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:09:44.456106 20531 raft_consensus.cc:729] T dc00a892c09141d8ab443a99a16db082 P 3793e3bed5e04acb8adbc620baf660a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3793e3bed5e04acb8adbc620baf660a4, State: Initialized, Role: FOLLOWER
I20260926 15:09:44.456442 20531 consensus_queue.cc:260] T dc00a892c09141d8ab443a99a16db082 P 3793e3bed5e04acb8adbc620baf660a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bffb811d93894ea5a14e6f4f56f4a196" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 44487 } } peers { permanent_uuid: "641358d46138484794ffdf3e0ceb4109" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 35581 } } peers { permanent_uuid: "3793e3bed5e04acb8adbc620baf660a4" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 45155 } }
I20260926 15:09:44.457031 20531 ts_tablet_manager.cc:1428] T dc00a892c09141d8ab443a99a16db082 P 3793e3bed5e04acb8adbc620baf660a4: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260926 15:09:44.661222 20537 raft_consensus.cc:482] T dc00a892c09141d8ab443a99a16db082 P 3793e3bed5e04acb8adbc620baf660a4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 15:09:44.661450 20537 raft_consensus.cc:504] T dc00a892c09141d8ab443a99a16db082 P 3793e3bed5e04acb8adbc620baf660a4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bffb811d93894ea5a14e6f4f56f4a196" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 44487 } } peers { permanent_uuid: "641358d46138484794ffdf3e0ceb4109" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 35581 } } peers { permanent_uuid: "3793e3bed5e04acb8adbc620baf660a4" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 45155 } }
I20260926 15:09:44.662129 20537 leader_election.cc:288] T dc00a892c09141d8ab443a99a16db082 P 3793e3bed5e04acb8adbc620baf660a4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bffb811d93894ea5a14e6f4f56f4a196 (127.12.34.193:44487), 641358d46138484794ffdf3e0ceb4109 (127.12.34.194:35581)
I20260926 15:09:44.662650 20178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc00a892c09141d8ab443a99a16db082" candidate_uuid: "3793e3bed5e04acb8adbc620baf660a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bffb811d93894ea5a14e6f4f56f4a196" is_pre_election: true
I20260926 15:09:44.662724 20322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc00a892c09141d8ab443a99a16db082" candidate_uuid: "3793e3bed5e04acb8adbc620baf660a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "641358d46138484794ffdf3e0ceb4109" is_pre_election: true
I20260926 15:09:44.662930 20178 raft_consensus.cc:2454] T dc00a892c09141d8ab443a99a16db082 P bffb811d93894ea5a14e6f4f56f4a196 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3793e3bed5e04acb8adbc620baf660a4 in term 0.
I20260926 15:09:44.662997 20322 raft_consensus.cc:2454] T dc00a892c09141d8ab443a99a16db082 P 641358d46138484794ffdf3e0ceb4109 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3793e3bed5e04acb8adbc620baf660a4 in term 0.
I20260926 15:09:44.663378 20390 leader_election.cc:302] T dc00a892c09141d8ab443a99a16db082 P 3793e3bed5e04acb8adbc620baf660a4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3793e3bed5e04acb8adbc620baf660a4, bffb811d93894ea5a14e6f4f56f4a196; no voters:
I20260926 15:09:44.663650 20537 raft_consensus.cc:2781] T dc00a892c09141d8ab443a99a16db082 P 3793e3bed5e04acb8adbc620baf660a4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 15:09:44.663779 20537 raft_consensus.cc:482] T dc00a892c09141d8ab443a99a16db082 P 3793e3bed5e04acb8adbc620baf660a4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 15:09:44.663897 20537 raft_consensus.cc:3037] T dc00a892c09141d8ab443a99a16db082 P 3793e3bed5e04acb8adbc620baf660a4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:09:44.666244 20537 raft_consensus.cc:504] T dc00a892c09141d8ab443a99a16db082 P 3793e3bed5e04acb8adbc620baf660a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bffb811d93894ea5a14e6f4f56f4a196" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 44487 } } peers { permanent_uuid: "641358d46138484794ffdf3e0ceb4109" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 35581 } } peers { permanent_uuid: "3793e3bed5e04acb8adbc620baf660a4" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 45155 } }
I20260926 15:09:44.666880 20537 leader_election.cc:288] T dc00a892c09141d8ab443a99a16db082 P 3793e3bed5e04acb8adbc620baf660a4 [CANDIDATE]: Term 1 election: Requested vote from peers bffb811d93894ea5a14e6f4f56f4a196 (127.12.34.193:44487), 641358d46138484794ffdf3e0ceb4109 (127.12.34.194:35581)
I20260926 15:09:44.667384 20322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc00a892c09141d8ab443a99a16db082" candidate_uuid: "3793e3bed5e04acb8adbc620baf660a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "641358d46138484794ffdf3e0ceb4109"
I20260926 15:09:44.667384 20178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc00a892c09141d8ab443a99a16db082" candidate_uuid: "3793e3bed5e04acb8adbc620baf660a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bffb811d93894ea5a14e6f4f56f4a196"
I20260926 15:09:44.667583 20178 raft_consensus.cc:3037] T dc00a892c09141d8ab443a99a16db082 P bffb811d93894ea5a14e6f4f56f4a196 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:09:44.667583 20322 raft_consensus.cc:3037] T dc00a892c09141d8ab443a99a16db082 P 641358d46138484794ffdf3e0ceb4109 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:09:44.670511 20178 raft_consensus.cc:2454] T dc00a892c09141d8ab443a99a16db082 P bffb811d93894ea5a14e6f4f56f4a196 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3793e3bed5e04acb8adbc620baf660a4 in term 1.
I20260926 15:09:44.670511 20322 raft_consensus.cc:2454] T dc00a892c09141d8ab443a99a16db082 P 641358d46138484794ffdf3e0ceb4109 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3793e3bed5e04acb8adbc620baf660a4 in term 1.
I20260926 15:09:44.670893 20388 leader_election.cc:302] T dc00a892c09141d8ab443a99a16db082 P 3793e3bed5e04acb8adbc620baf660a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3793e3bed5e04acb8adbc620baf660a4, 641358d46138484794ffdf3e0ceb4109; no voters:
I20260926 15:09:44.671200 20537 raft_consensus.cc:2781] T dc00a892c09141d8ab443a99a16db082 P 3793e3bed5e04acb8adbc620baf660a4 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:09:44.671367 20537 raft_consensus.cc:686] T dc00a892c09141d8ab443a99a16db082 P 3793e3bed5e04acb8adbc620baf660a4 [term 1 LEADER]: Becoming Leader. State: Replica: 3793e3bed5e04acb8adbc620baf660a4, State: Running, Role: LEADER
I20260926 15:09:44.671628 20537 consensus_queue.cc:237] T dc00a892c09141d8ab443a99a16db082 P 3793e3bed5e04acb8adbc620baf660a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bffb811d93894ea5a14e6f4f56f4a196" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 44487 } } peers { permanent_uuid: "641358d46138484794ffdf3e0ceb4109" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 35581 } } peers { permanent_uuid: "3793e3bed5e04acb8adbc620baf660a4" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 45155 } }
I20260926 15:09:44.674350 20020 catalog_manager.cc:5495] T dc00a892c09141d8ab443a99a16db082 P 3793e3bed5e04acb8adbc620baf660a4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3793e3bed5e04acb8adbc620baf660a4 (127.12.34.195). New cstate: current_term: 1 leader_uuid: "3793e3bed5e04acb8adbc620baf660a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bffb811d93894ea5a14e6f4f56f4a196" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 44487 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "641358d46138484794ffdf3e0ceb4109" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 35581 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3793e3bed5e04acb8adbc620baf660a4" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 45155 } health_report { overall_health: HEALTHY } } }
Sep 26 15:09:44 dist-test-slave-865v krb5kdc[19976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435381, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.34.254@KRBTEST.COM for oryx/127.12.34.254@KRBTEST.COM
I20260926 15:09:44.753405 20569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:44.739615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43074 (local address 127.12.34.254:46481)
0926 15:09:44.740567 (+ 952us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:44.740575 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:44.740598 (+ 23us) server_negotiation.cc:407] Connection header received
0926 15:09:44.740650 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:44.740653 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:44.740700 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:44.740820 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:09:44.741875 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:44.742636 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:44.743858 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:44.744307 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:44.744449 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:44.748153 (+ 3704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:09:44.748158 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:09:44.748162 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:09:44.748187 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:09:44.750815 (+ 2628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:44.751482 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:44.751490 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:44.751494 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:44.751609 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:44.751942 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:44.751949 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:44.751952 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:44.752210 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:09:44.752354 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0926 15:09:44.752724 (+ 370us) server_negotiation.cc:299] Negotiation successful
0926 15:09:44.752961 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":827,"thread_start_us":683,"threads_started":1}
I20260926 15:09:44.753489 20568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:44.739601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:46481 (local address 127.0.0.1:43074)
0926 15:09:44.740316 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0926 15:09:44.740342 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0926 15:09:44.740498 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:09:44.740829 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:09:44.740833 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:09:44.741553 (+ 720us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:09:44.741733 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:44.741741 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:44.742759 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:44.742767 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:44.743687 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:44.743697 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:44.744468 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:44.744473 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:44.744657 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:44.745481 (+ 824us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:09:44.745510 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:09:44.747946 (+ 2436us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:09:44.750960 (+ 3014us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:09:44.750971 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:09:44.750979 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:09:44.751368 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:09:44.751710 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:09:44.751717 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:09:44.751720 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:09:44.751846 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:09:44.752338 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:09:44.752345 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:09:44.752497 (+ 152us) client_negotiation.cc:769] Sending connection context
0926 15:09:44.752656 (+ 159us) client_negotiation.cc:240] Negotiation successful
0926 15:09:44.753065 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"mutex_wait_us":163,"thread_start_us":365,"threads_started":1}
I20260926 15:09:44.759298 20020 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.12.34.254@KRBTEST.COM'} at 127.0.0.1:43074:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 15:09:44.760308 20020 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 15:09:44.767978 20296 tablet_service.cc:1467] Processing CreateTablet for tablet fe42e3feded54f55b5df8d9cdacae051 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f9f65bdb1c8945f095966de633d85016]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 15:09:44.768217 20158 tablet_service.cc:1467] Processing CreateTablet for tablet fe42e3feded54f55b5df8d9cdacae051 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f9f65bdb1c8945f095966de633d85016]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 15:09:44.768558 20296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe42e3feded54f55b5df8d9cdacae051. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:09:44.768761 20158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe42e3feded54f55b5df8d9cdacae051. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:09:44.773851 20440 tablet_service.cc:1467] Processing CreateTablet for tablet fe42e3feded54f55b5df8d9cdacae051 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f9f65bdb1c8945f095966de633d85016]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 15:09:44.774366 20440 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe42e3feded54f55b5df8d9cdacae051. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:09:44.775856 20532 tablet_bootstrap.cc:492] T fe42e3feded54f55b5df8d9cdacae051 P 641358d46138484794ffdf3e0ceb4109: Bootstrap starting.
I20260926 15:09:44.777415 20532 tablet_bootstrap.cc:654] T fe42e3feded54f55b5df8d9cdacae051 P 641358d46138484794ffdf3e0ceb4109: Neither blocks nor log segments found. Creating new log.
I20260926 15:09:44.782204 20531 tablet_bootstrap.cc:492] T fe42e3feded54f55b5df8d9cdacae051 P 3793e3bed5e04acb8adbc620baf660a4: Bootstrap starting.
I20260926 15:09:44.783847 20531 tablet_bootstrap.cc:654] T fe42e3feded54f55b5df8d9cdacae051 P 3793e3bed5e04acb8adbc620baf660a4: Neither blocks nor log segments found. Creating new log.
I20260926 15:09:44.787374 20533 tablet_bootstrap.cc:492] T fe42e3feded54f55b5df8d9cdacae051 P bffb811d93894ea5a14e6f4f56f4a196: Bootstrap starting.
I20260926 15:09:44.789001 20533 tablet_bootstrap.cc:654] T fe42e3feded54f55b5df8d9cdacae051 P bffb811d93894ea5a14e6f4f56f4a196: Neither blocks nor log segments found. Creating new log.
I20260926 15:09:44.798964 20533 tablet_bootstrap.cc:492] T fe42e3feded54f55b5df8d9cdacae051 P bffb811d93894ea5a14e6f4f56f4a196: No bootstrap required, opened a new log
I20260926 15:09:44.799191 20533 ts_tablet_manager.cc:1397] T fe42e3feded54f55b5df8d9cdacae051 P bffb811d93894ea5a14e6f4f56f4a196: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.005s
I20260926 15:09:44.799950 20533 raft_consensus.cc:348] T fe42e3feded54f55b5df8d9cdacae051 P bffb811d93894ea5a14e6f4f56f4a196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bffb811d93894ea5a14e6f4f56f4a196" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 44487 } } peers { permanent_uuid: "641358d46138484794ffdf3e0ceb4109" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 35581 } } peers { permanent_uuid: "3793e3bed5e04acb8adbc620baf660a4" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 45155 } }
I20260926 15:09:44.800220 20533 raft_consensus.cc:374] T fe42e3feded54f55b5df8d9cdacae051 P bffb811d93894ea5a14e6f4f56f4a196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:09:44.800297 20533 raft_consensus.cc:729] T fe42e3feded54f55b5df8d9cdacae051 P bffb811d93894ea5a14e6f4f56f4a196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bffb811d93894ea5a14e6f4f56f4a196, State: Initialized, Role: FOLLOWER
I20260926 15:09:44.800585 20533 consensus_queue.cc:260] T fe42e3feded54f55b5df8d9cdacae051 P bffb811d93894ea5a14e6f4f56f4a196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bffb811d93894ea5a14e6f4f56f4a196" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 44487 } } peers { permanent_uuid: "641358d46138484794ffdf3e0ceb4109" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 35581 } } peers { permanent_uuid: "3793e3bed5e04acb8adbc620baf660a4" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 45155 } }
I20260926 15:09:44.801151 20533 ts_tablet_manager.cc:1428] T fe42e3feded54f55b5df8d9cdacae051 P bffb811d93894ea5a14e6f4f56f4a196: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260926 15:09:44.802183 20532 tablet_bootstrap.cc:492] T fe42e3feded54f55b5df8d9cdacae051 P 641358d46138484794ffdf3e0ceb4109: No bootstrap required, opened a new log
I20260926 15:09:44.802330 20532 ts_tablet_manager.cc:1397] T fe42e3feded54f55b5df8d9cdacae051 P 641358d46138484794ffdf3e0ceb4109: Time spent bootstrapping tablet: real 0.027s user 0.002s sys 0.009s
I20260926 15:09:44.802984 20532 raft_consensus.cc:348] T fe42e3feded54f55b5df8d9cdacae051 P 641358d46138484794ffdf3e0ceb4109 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bffb811d93894ea5a14e6f4f56f4a196" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 44487 } } peers { permanent_uuid: "641358d46138484794ffdf3e0ceb4109" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 35581 } } peers { permanent_uuid: "3793e3bed5e04acb8adbc620baf660a4" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 45155 } }
I20260926 15:09:44.803197 20532 raft_consensus.cc:374] T fe42e3feded54f55b5df8d9cdacae051 P 641358d46138484794ffdf3e0ceb4109 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:09:44.803254 20532 raft_consensus.cc:729] T fe42e3feded54f55b5df8d9cdacae051 P 641358d46138484794ffdf3e0ceb4109 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 641358d46138484794ffdf3e0ceb4109, State: Initialized, Role: FOLLOWER
I20260926 15:09:44.803478 20532 consensus_queue.cc:260] T fe42e3feded54f55b5df8d9cdacae051 P 641358d46138484794ffdf3e0ceb4109 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bffb811d93894ea5a14e6f4f56f4a196" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 44487 } } peers { permanent_uuid: "641358d46138484794ffdf3e0ceb4109" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 35581 } } peers { permanent_uuid: "3793e3bed5e04acb8adbc620baf660a4" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 45155 } }
I20260926 15:09:44.804049 20532 ts_tablet_manager.cc:1428] T fe42e3feded54f55b5df8d9cdacae051 P 641358d46138484794ffdf3e0ceb4109: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260926 15:09:44.803957 20538 tablet_replica.cc:439] T fe42e3feded54f55b5df8d9cdacae051 P 641358d46138484794ffdf3e0ceb4109: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bffb811d93894ea5a14e6f4f56f4a196" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 44487 } } peers { permanent_uuid: "641358d46138484794ffdf3e0ceb4109" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 35581 } } peers { permanent_uuid: "3793e3bed5e04acb8adbc620baf660a4" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 45155 } } }
I20260926 15:09:44.804344 20538 tablet_replica.cc:442] T fe42e3feded54f55b5df8d9cdacae051 P 641358d46138484794ffdf3e0ceb4109: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 15:09:44.804752 20531 tablet_bootstrap.cc:492] T fe42e3feded54f55b5df8d9cdacae051 P 3793e3bed5e04acb8adbc620baf660a4: No bootstrap required, opened a new log
I20260926 15:09:44.804894 20531 ts_tablet_manager.cc:1397] T fe42e3feded54f55b5df8d9cdacae051 P 3793e3bed5e04acb8adbc620baf660a4: Time spent bootstrapping tablet: real 0.023s user 0.006s sys 0.006s
I20260926 15:09:44.805017 20539 tablet_replica.cc:439] T fe42e3feded54f55b5df8d9cdacae051 P bffb811d93894ea5a14e6f4f56f4a196: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bffb811d93894ea5a14e6f4f56f4a196" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 44487 } } peers { permanent_uuid: "641358d46138484794ffdf3e0ceb4109" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 35581 } } peers { permanent_uuid: "3793e3bed5e04acb8adbc620baf660a4" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 45155 } } }
I20260926 15:09:44.805351 20539 tablet_replica.cc:442] T fe42e3feded54f55b5df8d9cdacae051 P bffb811d93894ea5a14e6f4f56f4a196: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 15:09:44.805693 20531 raft_consensus.cc:348] T fe42e3feded54f55b5df8d9cdacae051 P 3793e3bed5e04acb8adbc620baf660a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bffb811d93894ea5a14e6f4f56f4a196" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 44487 } } peers { permanent_uuid: "641358d46138484794ffdf3e0ceb4109" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 35581 } } peers { permanent_uuid: "3793e3bed5e04acb8adbc620baf660a4" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 45155 } }
I20260926 15:09:44.805953 20531 raft_consensus.cc:374] T fe42e3feded54f55b5df8d9cdacae051 P 3793e3bed5e04acb8adbc620baf660a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:09:44.806033 20531 raft_consensus.cc:729] T fe42e3feded54f55b5df8d9cdacae051 P 3793e3bed5e04acb8adbc620baf660a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3793e3bed5e04acb8adbc620baf660a4, State: Initialized, Role: FOLLOWER
I20260926 15:09:44.806315 20531 consensus_queue.cc:260] T fe42e3feded54f55b5df8d9cdacae051 P 3793e3bed5e04acb8adbc620baf660a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bffb811d93894ea5a14e6f4f56f4a196" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 44487 } } peers { permanent_uuid: "641358d46138484794ffdf3e0ceb4109" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 35581 } } peers { permanent_uuid: "3793e3bed5e04acb8adbc620baf660a4" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 45155 } }
I20260926 15:09:44.806738 20537 tablet_replica.cc:439] T fe42e3feded54f55b5df8d9cdacae051 P 3793e3bed5e04acb8adbc620baf660a4: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bffb811d93894ea5a14e6f4f56f4a196" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 44487 } } peers { permanent_uuid: "641358d46138484794ffdf3e0ceb4109" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 35581 } } peers { permanent_uuid: "3793e3bed5e04acb8adbc620baf660a4" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 45155 } } }
I20260926 15:09:44.807089 20537 tablet_replica.cc:442] T fe42e3feded54f55b5df8d9cdacae051 P 3793e3bed5e04acb8adbc620baf660a4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 15:09:44.808933 20531 ts_tablet_manager.cc:1428] T fe42e3feded54f55b5df8d9cdacae051 P 3793e3bed5e04acb8adbc620baf660a4: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20260926 15:09:44.811898 20539 raft_consensus.cc:482] T fe42e3feded54f55b5df8d9cdacae051 P bffb811d93894ea5a14e6f4f56f4a196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 15:09:44.812091 20539 raft_consensus.cc:504] T fe42e3feded54f55b5df8d9cdacae051 P bffb811d93894ea5a14e6f4f56f4a196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bffb811d93894ea5a14e6f4f56f4a196" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 44487 } } peers { permanent_uuid: "641358d46138484794ffdf3e0ceb4109" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 35581 } } peers { permanent_uuid: "3793e3bed5e04acb8adbc620baf660a4" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 45155 } }
I20260926 15:09:44.813714 20539 leader_election.cc:288] T fe42e3feded54f55b5df8d9cdacae051 P bffb811d93894ea5a14e6f4f56f4a196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 641358d46138484794ffdf3e0ceb4109 (127.12.34.194:35581), 3793e3bed5e04acb8adbc620baf660a4 (127.12.34.195:45155)
I20260926 15:09:44.841043 20526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:44.814027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.193:35749 (local address 127.12.34.195:45155)
0926 15:09:44.814199 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:44.814209 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:44.826460 (+ 12251us) server_negotiation.cc:407] Connection header received
0926 15:09:44.826676 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:44.826684 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:44.826781 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:44.826886 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:09:44.832321 (+ 5435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:44.833818 (+ 1497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:44.838423 (+ 4605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:44.839816 (+ 1393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:44.839946 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:44.840022 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 15:09:44.840541 (+ 519us) server_negotiation.cc:299] Negotiation successful
0926 15:09:44.840685 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 15:09:44.842020 20574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:44.813958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.195:45155 (local address 127.12.34.193:35749)
0926 15:09:44.826347 (+ 12389us) negotiation.cc:107] Waiting for socket to connect
0926 15:09:44.826375 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0926 15:09:44.826543 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:09:44.830853 (+ 4310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:09:44.830862 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:09:44.830903 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:09:44.831361 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:44.831377 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:44.834050 (+ 2673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:44.834056 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:44.836692 (+ 2636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:44.836714 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:44.840154 (+ 3440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:44.840160 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:44.840344 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:44.840384 (+ 40us) client_negotiation.cc:769] Sending connection context
0926 15:09:44.841534 (+ 1150us) client_negotiation.cc:240] Negotiation successful
0926 15:09:44.841678 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12201,"thread_start_us":23847,"threads_started":1}
I20260926 15:09:44.843051 20461 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe42e3feded54f55b5df8d9cdacae051" candidate_uuid: "bffb811d93894ea5a14e6f4f56f4a196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3793e3bed5e04acb8adbc620baf660a4" is_pre_election: true
I20260926 15:09:44.843472 20461 raft_consensus.cc:2454] T fe42e3feded54f55b5df8d9cdacae051 P 3793e3bed5e04acb8adbc620baf660a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bffb811d93894ea5a14e6f4f56f4a196 in term 0.
I20260926 15:09:44.844129 20108 leader_election.cc:302] T fe42e3feded54f55b5df8d9cdacae051 P bffb811d93894ea5a14e6f4f56f4a196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3793e3bed5e04acb8adbc620baf660a4, bffb811d93894ea5a14e6f4f56f4a196; no voters:
I20260926 15:09:44.844630 20539 raft_consensus.cc:2781] T fe42e3feded54f55b5df8d9cdacae051 P bffb811d93894ea5a14e6f4f56f4a196 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 15:09:44.844758 20539 raft_consensus.cc:482] T fe42e3feded54f55b5df8d9cdacae051 P bffb811d93894ea5a14e6f4f56f4a196 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 15:09:44.844842 20539 raft_consensus.cc:3037] T fe42e3feded54f55b5df8d9cdacae051 P bffb811d93894ea5a14e6f4f56f4a196 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:09:44.845264 20575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:44.814917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.193:37811 (local address 127.12.34.194:35581)
0926 15:09:44.837556 (+ 22639us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:44.837565 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:44.838188 (+ 623us) server_negotiation.cc:407] Connection header received
0926 15:09:44.838393 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:44.838399 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:44.838475 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:44.838565 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:09:44.839327 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:44.840489 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:44.842835 (+ 2346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:44.843701 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:44.843865 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:44.843935 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 15:09:44.844693 (+ 758us) server_negotiation.cc:299] Negotiation successful
0926 15:09:44.844859 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22449,"thread_start_us":22661,"threads_started":1}
I20260926 15:09:44.847584 20539 raft_consensus.cc:504] T fe42e3feded54f55b5df8d9cdacae051 P bffb811d93894ea5a14e6f4f56f4a196 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bffb811d93894ea5a14e6f4f56f4a196" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 44487 } } peers { permanent_uuid: "641358d46138484794ffdf3e0ceb4109" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 35581 } } peers { permanent_uuid: "3793e3bed5e04acb8adbc620baf660a4" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 45155 } }
I20260926 15:09:44.848140 20539 leader_election.cc:288] T fe42e3feded54f55b5df8d9cdacae051 P bffb811d93894ea5a14e6f4f56f4a196 [CANDIDATE]: Term 1 election: Requested vote from peers 641358d46138484794ffdf3e0ceb4109 (127.12.34.194:35581), 3793e3bed5e04acb8adbc620baf660a4 (127.12.34.195:45155)
I20260926 15:09:44.848786 20573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:44.814544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.194:35581 (local address 127.12.34.193:37811)
0926 15:09:44.838070 (+ 23526us) negotiation.cc:107] Waiting for socket to connect
0926 15:09:44.838087 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 15:09:44.838236 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:09:44.838733 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:09:44.838740 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:09:44.838779 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:09:44.839101 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:44.839113 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:44.840626 (+ 1513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:44.840631 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:44.842643 (+ 2012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:44.842655 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:44.844152 (+ 1497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:44.844165 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:44.844339 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:44.844375 (+ 36us) client_negotiation.cc:769] Sending connection context
0926 15:09:44.844473 (+ 98us) client_negotiation.cc:240] Negotiation successful
0926 15:09:44.844599 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23400,"thread_start_us":11471,"threads_started":1}
I20260926 15:09:44.848920 20461 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe42e3feded54f55b5df8d9cdacae051" candidate_uuid: "bffb811d93894ea5a14e6f4f56f4a196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3793e3bed5e04acb8adbc620baf660a4"
I20260926 15:09:44.849159 20461 raft_consensus.cc:3037] T fe42e3feded54f55b5df8d9cdacae051 P 3793e3bed5e04acb8adbc620baf660a4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:09:44.849458 20322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe42e3feded54f55b5df8d9cdacae051" candidate_uuid: "bffb811d93894ea5a14e6f4f56f4a196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "641358d46138484794ffdf3e0ceb4109" is_pre_election: true
I20260926 15:09:44.849722 20322 raft_consensus.cc:2454] T fe42e3feded54f55b5df8d9cdacae051 P 641358d46138484794ffdf3e0ceb4109 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bffb811d93894ea5a14e6f4f56f4a196 in term 0.
I20260926 15:09:44.849671 20321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe42e3feded54f55b5df8d9cdacae051" candidate_uuid: "bffb811d93894ea5a14e6f4f56f4a196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "641358d46138484794ffdf3e0ceb4109"
I20260926 15:09:44.849906 20321 raft_consensus.cc:3037] T fe42e3feded54f55b5df8d9cdacae051 P 641358d46138484794ffdf3e0ceb4109 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:09:44.852312 20461 raft_consensus.cc:2454] T fe42e3feded54f55b5df8d9cdacae051 P 3793e3bed5e04acb8adbc620baf660a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bffb811d93894ea5a14e6f4f56f4a196 in term 1.
I20260926 15:09:44.852669 20108 leader_election.cc:302] T fe42e3feded54f55b5df8d9cdacae051 P bffb811d93894ea5a14e6f4f56f4a196 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3793e3bed5e04acb8adbc620baf660a4, bffb811d93894ea5a14e6f4f56f4a196; no voters:
I20260926 15:09:44.852862 20321 raft_consensus.cc:2454] T fe42e3feded54f55b5df8d9cdacae051 P 641358d46138484794ffdf3e0ceb4109 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bffb811d93894ea5a14e6f4f56f4a196 in term 1.
I20260926 15:09:44.852958 20539 raft_consensus.cc:2781] T fe42e3feded54f55b5df8d9cdacae051 P bffb811d93894ea5a14e6f4f56f4a196 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:09:44.853668 20539 raft_consensus.cc:686] T fe42e3feded54f55b5df8d9cdacae051 P bffb811d93894ea5a14e6f4f56f4a196 [term 1 LEADER]: Becoming Leader. State: Replica: bffb811d93894ea5a14e6f4f56f4a196, State: Running, Role: LEADER
I20260926 15:09:44.854348 20539 consensus_queue.cc:237] T fe42e3feded54f55b5df8d9cdacae051 P bffb811d93894ea5a14e6f4f56f4a196 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bffb811d93894ea5a14e6f4f56f4a196" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 44487 } } peers { permanent_uuid: "641358d46138484794ffdf3e0ceb4109" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 35581 } } peers { permanent_uuid: "3793e3bed5e04acb8adbc620baf660a4" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 45155 } }
I20260926 15:09:44.856310 20576 tablet_replica.cc:439] T fe42e3feded54f55b5df8d9cdacae051 P bffb811d93894ea5a14e6f4f56f4a196: TxnStatusTablet state changed. Reason: New leader bffb811d93894ea5a14e6f4f56f4a196. Latest consensus state: current_term: 1 leader_uuid: "bffb811d93894ea5a14e6f4f56f4a196" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bffb811d93894ea5a14e6f4f56f4a196" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 44487 } } peers { permanent_uuid: "641358d46138484794ffdf3e0ceb4109" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 35581 } } peers { permanent_uuid: "3793e3bed5e04acb8adbc620baf660a4" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 45155 } } }
I20260926 15:09:44.856664 20576 tablet_replica.cc:442] T fe42e3feded54f55b5df8d9cdacae051 P bffb811d93894ea5a14e6f4f56f4a196: This TxnStatusTablet replica's current role is: LEADER
I20260926 15:09:44.857676 20578 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 15:09:44.858153 20578 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 15:09:44.858947 20020 catalog_manager.cc:5495] T fe42e3feded54f55b5df8d9cdacae051 P bffb811d93894ea5a14e6f4f56f4a196 reported cstate change: term changed from 0 to 1, leader changed from <none> to bffb811d93894ea5a14e6f4f56f4a196 (127.12.34.193). New cstate: current_term: 1 leader_uuid: "bffb811d93894ea5a14e6f4f56f4a196" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bffb811d93894ea5a14e6f4f56f4a196" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 44487 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "641358d46138484794ffdf3e0ceb4109" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 35581 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3793e3bed5e04acb8adbc620baf660a4" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 45155 } health_report { overall_health: UNKNOWN } } }
I20260926 15:09:44.877616 20568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:44.870807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.193:44487 (local address 127.0.0.1:44036)
0926 15:09:44.871075 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0926 15:09:44.871102 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0926 15:09:44.871251 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:09:44.871848 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:09:44.871853 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:09:44.871863 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 15:09:44.872012 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:44.872019 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:44.873593 (+ 1574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:44.873598 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:44.874711 (+ 1113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:44.874724 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:44.875675 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:44.875679 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:44.875807 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:44.875864 (+ 57us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 15:09:44.876969 (+ 1105us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 15:09:44.876988 (+ 19us) client_negotiation.cc:769] Sending connection context
0926 15:09:44.877070 (+ 82us) client_negotiation.cc:240] Negotiation successful
0926 15:09:44.877289 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260926 15:09:44.877660 20581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:44.870792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44036 (local address 127.12.34.193:44487)
0926 15:09:44.871440 (+ 648us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:44.871447 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:44.871471 (+ 24us) server_negotiation.cc:407] Connection header received
0926 15:09:44.871552 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:44.871557 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:44.871650 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:44.871807 (+ 157us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 15:09:44.872184 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:44.873456 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:44.874854 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:44.875529 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:44.875652 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:44.876097 (+ 445us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 15:09:44.876802 (+ 705us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 15:09:44.876950 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0926 15:09:44.877144 (+ 194us) server_negotiation.cc:299] Negotiation successful
0926 15:09:44.877262 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":521,"thread_start_us":373,"threads_started":1}
I20260926 15:09:44.881757 20321 raft_consensus.cc:1261] T fe42e3feded54f55b5df8d9cdacae051 P 641358d46138484794ffdf3e0ceb4109 [term 1 FOLLOWER]: Refusing update from remote peer bffb811d93894ea5a14e6f4f56f4a196: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:09:44.881996 20461 raft_consensus.cc:1261] T fe42e3feded54f55b5df8d9cdacae051 P 3793e3bed5e04acb8adbc620baf660a4 [term 1 FOLLOWER]: Refusing update from remote peer bffb811d93894ea5a14e6f4f56f4a196: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:09:44.882396 20576 consensus_queue.cc:1035] T fe42e3feded54f55b5df8d9cdacae051 P bffb811d93894ea5a14e6f4f56f4a196 [LEADER]: Connected to new peer: Peer: permanent_uuid: "641358d46138484794ffdf3e0ceb4109" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 35581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:09:44.882719 20539 consensus_queue.cc:1035] T fe42e3feded54f55b5df8d9cdacae051 P bffb811d93894ea5a14e6f4f56f4a196 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3793e3bed5e04acb8adbc620baf660a4" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 45155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:09:44.887398 20538 tablet_replica.cc:439] T fe42e3feded54f55b5df8d9cdacae051 P 641358d46138484794ffdf3e0ceb4109: TxnStatusTablet state changed. Reason: New leader bffb811d93894ea5a14e6f4f56f4a196. Latest consensus state: current_term: 1 leader_uuid: "bffb811d93894ea5a14e6f4f56f4a196" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bffb811d93894ea5a14e6f4f56f4a196" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 44487 } } peers { permanent_uuid: "641358d46138484794ffdf3e0ceb4109" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 35581 } } peers { permanent_uuid: "3793e3bed5e04acb8adbc620baf660a4" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 45155 } } }
I20260926 15:09:44.887676 20538 tablet_replica.cc:442] T fe42e3feded54f55b5df8d9cdacae051 P 641358d46138484794ffdf3e0ceb4109: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 15:09:44.888684 20576 tablet_replica.cc:439] T fe42e3feded54f55b5df8d9cdacae051 P bffb811d93894ea5a14e6f4f56f4a196: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bffb811d93894ea5a14e6f4f56f4a196" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bffb811d93894ea5a14e6f4f56f4a196" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 44487 } } peers { permanent_uuid: "641358d46138484794ffdf3e0ceb4109" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 35581 } } peers { permanent_uuid: "3793e3bed5e04acb8adbc620baf660a4" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 45155 } } }
I20260926 15:09:44.888902 20576 tablet_replica.cc:442] T fe42e3feded54f55b5df8d9cdacae051 P bffb811d93894ea5a14e6f4f56f4a196: This TxnStatusTablet replica's current role is: LEADER
I20260926 15:09:44.890348 20538 tablet_replica.cc:439] T fe42e3feded54f55b5df8d9cdacae051 P 641358d46138484794ffdf3e0ceb4109: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bffb811d93894ea5a14e6f4f56f4a196" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bffb811d93894ea5a14e6f4f56f4a196" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 44487 } } peers { permanent_uuid: "641358d46138484794ffdf3e0ceb4109" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 35581 } } peers { permanent_uuid: "3793e3bed5e04acb8adbc620baf660a4" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 45155 } } }
I20260926 15:09:44.890539 20538 tablet_replica.cc:442] T fe42e3feded54f55b5df8d9cdacae051 P 641358d46138484794ffdf3e0ceb4109: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 15:09:44.898434 20537 tablet_replica.cc:439] T fe42e3feded54f55b5df8d9cdacae051 P 3793e3bed5e04acb8adbc620baf660a4: TxnStatusTablet state changed. Reason: New leader bffb811d93894ea5a14e6f4f56f4a196. Latest consensus state: current_term: 1 leader_uuid: "bffb811d93894ea5a14e6f4f56f4a196" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bffb811d93894ea5a14e6f4f56f4a196" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 44487 } } peers { permanent_uuid: "641358d46138484794ffdf3e0ceb4109" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 35581 } } peers { permanent_uuid: "3793e3bed5e04acb8adbc620baf660a4" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 45155 } } }
I20260926 15:09:44.898705 20537 tablet_replica.cc:442] T fe42e3feded54f55b5df8d9cdacae051 P 3793e3bed5e04acb8adbc620baf660a4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 15:09:44.900946 20539 tablet_replica.cc:439] T fe42e3feded54f55b5df8d9cdacae051 P bffb811d93894ea5a14e6f4f56f4a196: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bffb811d93894ea5a14e6f4f56f4a196" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bffb811d93894ea5a14e6f4f56f4a196" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 44487 } } peers { permanent_uuid: "641358d46138484794ffdf3e0ceb4109" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 35581 } } peers { permanent_uuid: "3793e3bed5e04acb8adbc620baf660a4" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 45155 } } }
I20260926 15:09:44.901239 20539 tablet_replica.cc:442] T fe42e3feded54f55b5df8d9cdacae051 P bffb811d93894ea5a14e6f4f56f4a196: This TxnStatusTablet replica's current role is: LEADER
I20260926 15:09:44.904454 20537 tablet_replica.cc:439] T fe42e3feded54f55b5df8d9cdacae051 P 3793e3bed5e04acb8adbc620baf660a4: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bffb811d93894ea5a14e6f4f56f4a196" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bffb811d93894ea5a14e6f4f56f4a196" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 44487 } } peers { permanent_uuid: "641358d46138484794ffdf3e0ceb4109" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 35581 } } peers { permanent_uuid: "3793e3bed5e04acb8adbc620baf660a4" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 45155 } } }
I20260926 15:09:44.904711 20537 tablet_replica.cc:442] T fe42e3feded54f55b5df8d9cdacae051 P 3793e3bed5e04acb8adbc620baf660a4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 15:09:44.918534 20581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:44.912641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.195:38417 (local address 127.12.34.193:44487)
0926 15:09:44.913005 (+ 364us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:44.913168 (+ 163us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:44.913705 (+ 537us) server_negotiation.cc:407] Connection header received
0926 15:09:44.913914 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:44.913920 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:44.913999 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:44.914105 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 15:09:44.914756 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:44.915485 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:44.916771 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:44.917289 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:44.917432 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:44.917827 (+ 395us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 15:09:44.917927 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 15:09:44.918036 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 15:09:44.918141 (+ 105us) server_negotiation.cc:299] Negotiation successful
0926 15:09:44.918227 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195}
I20260926 15:09:44.918622 20589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:44.912513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.193:44487 (local address 127.12.34.195:38417)
0926 15:09:44.913586 (+ 1073us) negotiation.cc:107] Waiting for socket to connect
0926 15:09:44.913603 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 15:09:44.913783 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:09:44.914126 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:09:44.914130 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:09:44.914141 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 15:09:44.914587 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:44.914596 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:44.915628 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:44.915636 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:44.916617 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:44.916628 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:44.917445 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:44.917451 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:44.917596 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:44.917646 (+ 50us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 15:09:44.918036 (+ 390us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 15:09:44.918049 (+ 13us) client_negotiation.cc:769] Sending connection context
0926 15:09:44.918124 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 15:09:44.918255 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":900,"thread_start_us":688,"threads_started":1}
I20260926 15:09:44.927124 20321 raft_consensus.cc:1261] T dc00a892c09141d8ab443a99a16db082 P 641358d46138484794ffdf3e0ceb4109 [term 1 FOLLOWER]: Refusing update from remote peer 3793e3bed5e04acb8adbc620baf660a4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:09:44.927315 20178 raft_consensus.cc:1261] T dc00a892c09141d8ab443a99a16db082 P bffb811d93894ea5a14e6f4f56f4a196 [term 1 FOLLOWER]: Refusing update from remote peer 3793e3bed5e04acb8adbc620baf660a4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:09:44.927752 20553 consensus_queue.cc:1035] T dc00a892c09141d8ab443a99a16db082 P 3793e3bed5e04acb8adbc620baf660a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "641358d46138484794ffdf3e0ceb4109" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 35581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:09:44.928248 20537 consensus_queue.cc:1035] T dc00a892c09141d8ab443a99a16db082 P 3793e3bed5e04acb8adbc620baf660a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bffb811d93894ea5a14e6f4f56f4a196" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 44487 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:09:44.959682 20526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:44.953853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.193:34307 (local address 127.12.34.195:45155)
0926 15:09:44.954059 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:44.954065 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:44.954505 (+ 440us) server_negotiation.cc:407] Connection header received
0926 15:09:44.954718 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:44.954723 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:44.954810 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:44.954905 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 15:09:44.955681 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:44.956381 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:44.957958 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:44.958361 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:44.958488 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:44.958933 (+ 445us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 15:09:44.959076 (+ 143us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 15:09:44.959166 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 15:09:44.959323 (+ 157us) server_negotiation.cc:299] Negotiation successful
0926 15:09:44.959405 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 15:09:44.959853 20594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:44.953777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.195:45155 (local address 127.12.34.193:34307)
0926 15:09:44.954400 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0926 15:09:44.954438 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 15:09:44.954581 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:09:44.954948 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:09:44.954953 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:09:44.954966 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 15:09:44.955458 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:44.955473 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:44.956507 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:44.956515 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:44.957816 (+ 1301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:44.957828 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:44.958547 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:44.958553 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:44.958703 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:44.958755 (+ 52us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 15:09:44.959184 (+ 429us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 15:09:44.959197 (+ 13us) client_negotiation.cc:769] Sending connection context
0926 15:09:44.959284 (+ 87us) client_negotiation.cc:240] Negotiation successful
0926 15:09:44.959422 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":257,"threads_started":1}
I20260926 15:09:44.990147 20021 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43066:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 15:09:44.990450 20021 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43066:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 15:09:44.993592 20021 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1a3d30fc883a4f67bc16c22629712a96: Sending DeleteTablet for 3 replicas of tablet dc00a892c09141d8ab443a99a16db082
I20260926 15:09:44.994283 20158 tablet_service.cc:1514] Processing DeleteTablet for tablet dc00a892c09141d8ab443a99a16db082 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:09:44 UTC) from {username='oryx', principal='oryx/127.12.34.254@KRBTEST.COM'} at 127.0.0.1:44032
I20260926 15:09:44.994406 20296 tablet_service.cc:1514] Processing DeleteTablet for tablet dc00a892c09141d8ab443a99a16db082 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:09:44 UTC) from {username='oryx', principal='oryx/127.12.34.254@KRBTEST.COM'} at 127.0.0.1:41168
I20260926 15:09:44.994521 20440 tablet_service.cc:1514] Processing DeleteTablet for tablet dc00a892c09141d8ab443a99a16db082 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:09:44 UTC) from {username='oryx', principal='oryx/127.12.34.254@KRBTEST.COM'} at 127.0.0.1:37212
I20260926 15:09:45.006296 20597 tablet_replica.cc:330] T dc00a892c09141d8ab443a99a16db082 P 641358d46138484794ffdf3e0ceb4109: stopping tablet replica
I20260926 15:09:45.006523 20598 tablet_replica.cc:330] T dc00a892c09141d8ab443a99a16db082 P 3793e3bed5e04acb8adbc620baf660a4: stopping tablet replica
I20260926 15:09:45.006527 20597 raft_consensus.cc:2229] T dc00a892c09141d8ab443a99a16db082 P 641358d46138484794ffdf3e0ceb4109 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 15:09:45.006742 20597 raft_consensus.cc:2258] T dc00a892c09141d8ab443a99a16db082 P 641358d46138484794ffdf3e0ceb4109 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:09:45.006794 20598 raft_consensus.cc:2229] T dc00a892c09141d8ab443a99a16db082 P 3793e3bed5e04acb8adbc620baf660a4 [term 1 LEADER]: Raft consensus shutting down.
I20260926 15:09:45.007266 20598 raft_consensus.cc:2258] T dc00a892c09141d8ab443a99a16db082 P 3793e3bed5e04acb8adbc620baf660a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:09:45.008107 20597 ts_tablet_manager.cc:1905] T dc00a892c09141d8ab443a99a16db082 P 641358d46138484794ffdf3e0ceb4109: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:09:45.008258 20598 ts_tablet_manager.cc:1905] T dc00a892c09141d8ab443a99a16db082 P 3793e3bed5e04acb8adbc620baf660a4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:09:45.015383 20598 ts_tablet_manager.cc:1918] T dc00a892c09141d8ab443a99a16db082 P 3793e3bed5e04acb8adbc620baf660a4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 15:09:45.015393 20597 ts_tablet_manager.cc:1918] T dc00a892c09141d8ab443a99a16db082 P 641358d46138484794ffdf3e0ceb4109: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 15:09:45.015511 20597 log.cc:1196] T dc00a892c09141d8ab443a99a16db082 P 641358d46138484794ffdf3e0ceb4109: Deleting WAL directory at /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435318278681-12427-0/minicluster-data/ts-1/wal/wals/dc00a892c09141d8ab443a99a16db082
I20260926 15:09:45.015678 20598 log.cc:1196] T dc00a892c09141d8ab443a99a16db082 P 3793e3bed5e04acb8adbc620baf660a4: Deleting WAL directory at /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435318278681-12427-0/minicluster-data/ts-2/wal/wals/dc00a892c09141d8ab443a99a16db082
I20260926 15:09:45.015954 20597 ts_tablet_manager.cc:1939] T dc00a892c09141d8ab443a99a16db082 P 641358d46138484794ffdf3e0ceb4109: Deleting consensus metadata
I20260926 15:09:45.016213 20598 ts_tablet_manager.cc:1939] T dc00a892c09141d8ab443a99a16db082 P 3793e3bed5e04acb8adbc620baf660a4: Deleting consensus metadata
I20260926 15:09:45.017069 20006 catalog_manager.cc:4841] TS 641358d46138484794ffdf3e0ceb4109 (127.12.34.194:35581): tablet dc00a892c09141d8ab443a99a16db082 (table test-table [id=fc77104f72c64155a07f00405d7fed6e]) successfully deleted
I20260926 15:09:45.017210 20005 catalog_manager.cc:4841] TS 3793e3bed5e04acb8adbc620baf660a4 (127.12.34.195:45155): tablet dc00a892c09141d8ab443a99a16db082 (table test-table [id=fc77104f72c64155a07f00405d7fed6e]) successfully deleted
I20260926 15:09:45.019474 20596 tablet_replica.cc:330] T dc00a892c09141d8ab443a99a16db082 P bffb811d93894ea5a14e6f4f56f4a196: stopping tablet replica
I20260926 15:09:45.019680 20596 raft_consensus.cc:2229] T dc00a892c09141d8ab443a99a16db082 P bffb811d93894ea5a14e6f4f56f4a196 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 15:09:45.019956 20596 raft_consensus.cc:2258] T dc00a892c09141d8ab443a99a16db082 P bffb811d93894ea5a14e6f4f56f4a196 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:09:45.021003 20596 ts_tablet_manager.cc:1905] T dc00a892c09141d8ab443a99a16db082 P bffb811d93894ea5a14e6f4f56f4a196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:09:45.028522 20596 ts_tablet_manager.cc:1918] T dc00a892c09141d8ab443a99a16db082 P bffb811d93894ea5a14e6f4f56f4a196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 15:09:45.028647 20596 log.cc:1196] T dc00a892c09141d8ab443a99a16db082 P bffb811d93894ea5a14e6f4f56f4a196: Deleting WAL directory at /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435318278681-12427-0/minicluster-data/ts-0/wal/wals/dc00a892c09141d8ab443a99a16db082
I20260926 15:09:45.029135 20596 ts_tablet_manager.cc:1939] T dc00a892c09141d8ab443a99a16db082 P bffb811d93894ea5a14e6f4f56f4a196: Deleting consensus metadata
I20260926 15:09:45.030220 20008 catalog_manager.cc:4841] TS bffb811d93894ea5a14e6f4f56f4a196 (127.12.34.193:44487): tablet dc00a892c09141d8ab443a99a16db082 (table test-table [id=fc77104f72c64155a07f00405d7fed6e]) successfully deleted
Sep 26 15:09:45 dist-test-slave-865v krb5kdc[19976](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435385, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 15:09:45 dist-test-slave-865v krb5kdc[19976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.12.34.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 15:09:45 dist-test-slave-865v krb5kdc[19976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.12.34.254@KRBTEST.COM, Server not found in Kerberos database
W20260926 15:09:45.063411 20569 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:09:45.052266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43088 (local address 127.12.34.254:46481)
0926 15:09:45.052477 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:45.052483 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:45.052839 (+ 356us) server_negotiation.cc:407] Connection header received
0926 15:09:45.052985 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:45.052991 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:45.053073 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:45.053201 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:09:45.054582 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:45.055712 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:45.056927 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:45.057595 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:45.057722 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:45.063158 (+ 5436us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43088: BlockingRecv error: recv got EOF from 127.0.0.1:43088 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 26 15:09:45 dist-test-slave-865v krb5kdc[19976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435385, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.12.34.254@KRBTEST.COM
I20260926 15:09:45.086526 20569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:45.069909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43092 (local address 127.12.34.254:46481)
0926 15:09:45.070278 (+ 369us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:45.070284 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:45.070656 (+ 372us) server_negotiation.cc:407] Connection header received
0926 15:09:45.070723 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:45.070727 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:45.070789 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:45.070883 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:09:45.072518 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:45.073721 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:45.075447 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:45.075842 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:45.075941 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:45.080346 (+ 4405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:09:45.080354 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:09:45.080359 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:09:45.080402 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:09:45.083824 (+ 3422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:45.084505 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:45.084511 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:45.084514 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:45.084687 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:45.085102 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:45.085108 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:45.085111 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:45.085439 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:09:45.085710 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0926 15:09:45.085942 (+ 232us) server_negotiation.cc:299] Negotiation successful
0926 15:09:45.086158 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260926 15:09:45.090348 20021 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43092:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 15:09:45.091075 20021 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 15:09:45.097956 20296 tablet_service.cc:1467] Processing CreateTablet for tablet 19aa73ba1b5f4d78869a29019cdc7853 (DEFAULT_TABLE table=test-table [id=ac99adce1e7e438498ecc0d7da93c5ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:09:45.097929 20158 tablet_service.cc:1467] Processing CreateTablet for tablet 19aa73ba1b5f4d78869a29019cdc7853 (DEFAULT_TABLE table=test-table [id=ac99adce1e7e438498ecc0d7da93c5ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:09:45.098369 20158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19aa73ba1b5f4d78869a29019cdc7853. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:09:45.098397 20296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19aa73ba1b5f4d78869a29019cdc7853. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:09:45.101814 20440 tablet_service.cc:1467] Processing CreateTablet for tablet 19aa73ba1b5f4d78869a29019cdc7853 (DEFAULT_TABLE table=test-table [id=ac99adce1e7e438498ecc0d7da93c5ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:09:45.102267 20440 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19aa73ba1b5f4d78869a29019cdc7853. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:09:45.104766 20532 tablet_bootstrap.cc:492] T 19aa73ba1b5f4d78869a29019cdc7853 P 641358d46138484794ffdf3e0ceb4109: Bootstrap starting.
I20260926 15:09:45.106456 20532 tablet_bootstrap.cc:654] T 19aa73ba1b5f4d78869a29019cdc7853 P 641358d46138484794ffdf3e0ceb4109: Neither blocks nor log segments found. Creating new log.
I20260926 15:09:45.110817 20533 tablet_bootstrap.cc:492] T 19aa73ba1b5f4d78869a29019cdc7853 P bffb811d93894ea5a14e6f4f56f4a196: Bootstrap starting.
I20260926 15:09:45.111469 20531 tablet_bootstrap.cc:492] T 19aa73ba1b5f4d78869a29019cdc7853 P 3793e3bed5e04acb8adbc620baf660a4: Bootstrap starting.
I20260926 15:09:45.112484 20533 tablet_bootstrap.cc:654] T 19aa73ba1b5f4d78869a29019cdc7853 P bffb811d93894ea5a14e6f4f56f4a196: Neither blocks nor log segments found. Creating new log.
I20260926 15:09:45.112895 20531 tablet_bootstrap.cc:654] T 19aa73ba1b5f4d78869a29019cdc7853 P 3793e3bed5e04acb8adbc620baf660a4: Neither blocks nor log segments found. Creating new log.
I20260926 15:09:45.113044 20532 tablet_bootstrap.cc:492] T 19aa73ba1b5f4d78869a29019cdc7853 P 641358d46138484794ffdf3e0ceb4109: No bootstrap required, opened a new log
I20260926 15:09:45.113234 20532 ts_tablet_manager.cc:1397] T 19aa73ba1b5f4d78869a29019cdc7853 P 641358d46138484794ffdf3e0ceb4109: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.004s
I20260926 15:09:45.114131 20532 raft_consensus.cc:348] T 19aa73ba1b5f4d78869a29019cdc7853 P 641358d46138484794ffdf3e0ceb4109 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bffb811d93894ea5a14e6f4f56f4a196" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 44487 } } peers { permanent_uuid: "641358d46138484794ffdf3e0ceb4109" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 35581 } } peers { permanent_uuid: "3793e3bed5e04acb8adbc620baf660a4" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 45155 } }
I20260926 15:09:45.114477 20532 raft_consensus.cc:374] T 19aa73ba1b5f4d78869a29019cdc7853 P 641358d46138484794ffdf3e0ceb4109 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:09:45.114588 20532 raft_consensus.cc:729] T 19aa73ba1b5f4d78869a29019cdc7853 P 641358d46138484794ffdf3e0ceb4109 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 641358d46138484794ffdf3e0ceb4109, State: Initialized, Role: FOLLOWER
I20260926 15:09:45.115043 20533 tablet_bootstrap.cc:492] T 19aa73ba1b5f4d78869a29019cdc7853 P bffb811d93894ea5a14e6f4f56f4a196: No bootstrap required, opened a new log
I20260926 15:09:45.114907 20532 consensus_queue.cc:260] T 19aa73ba1b5f4d78869a29019cdc7853 P 641358d46138484794ffdf3e0ceb4109 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bffb811d93894ea5a14e6f4f56f4a196" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 44487 } } peers { permanent_uuid: "641358d46138484794ffdf3e0ceb4109" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 35581 } } peers { permanent_uuid: "3793e3bed5e04acb8adbc620baf660a4" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 45155 } }
I20260926 15:09:45.115191 20533 ts_tablet_manager.cc:1397] T 19aa73ba1b5f4d78869a29019cdc7853 P bffb811d93894ea5a14e6f4f56f4a196: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260926 15:09:45.115760 20533 raft_consensus.cc:348] T 19aa73ba1b5f4d78869a29019cdc7853 P bffb811d93894ea5a14e6f4f56f4a196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bffb811d93894ea5a14e6f4f56f4a196" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 44487 } } peers { permanent_uuid: "641358d46138484794ffdf3e0ceb4109" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 35581 } } peers { permanent_uuid: "3793e3bed5e04acb8adbc620baf660a4" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 45155 } }
I20260926 15:09:45.116087 20533 raft_consensus.cc:374] T 19aa73ba1b5f4d78869a29019cdc7853 P bffb811d93894ea5a14e6f4f56f4a196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:09:45.116199 20533 raft_consensus.cc:729] T 19aa73ba1b5f4d78869a29019cdc7853 P bffb811d93894ea5a14e6f4f56f4a196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bffb811d93894ea5a14e6f4f56f4a196, State: Initialized, Role: FOLLOWER
I20260926 15:09:45.116328 20532 ts_tablet_manager.cc:1428] T 19aa73ba1b5f4d78869a29019cdc7853 P 641358d46138484794ffdf3e0ceb4109: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260926 15:09:45.116564 20533 consensus_queue.cc:260] T 19aa73ba1b5f4d78869a29019cdc7853 P bffb811d93894ea5a14e6f4f56f4a196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bffb811d93894ea5a14e6f4f56f4a196" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 44487 } } peers { permanent_uuid: "641358d46138484794ffdf3e0ceb4109" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 35581 } } peers { permanent_uuid: "3793e3bed5e04acb8adbc620baf660a4" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 45155 } }
I20260926 15:09:45.117884 20531 tablet_bootstrap.cc:492] T 19aa73ba1b5f4d78869a29019cdc7853 P 3793e3bed5e04acb8adbc620baf660a4: No bootstrap required, opened a new log
I20260926 15:09:45.118077 20531 ts_tablet_manager.cc:1397] T 19aa73ba1b5f4d78869a29019cdc7853 P 3793e3bed5e04acb8adbc620baf660a4: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.004s
I20260926 15:09:45.118873 20531 raft_consensus.cc:348] T 19aa73ba1b5f4d78869a29019cdc7853 P 3793e3bed5e04acb8adbc620baf660a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bffb811d93894ea5a14e6f4f56f4a196" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 44487 } } peers { permanent_uuid: "641358d46138484794ffdf3e0ceb4109" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 35581 } } peers { permanent_uuid: "3793e3bed5e04acb8adbc620baf660a4" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 45155 } }
I20260926 15:09:45.119154 20531 raft_consensus.cc:374] T 19aa73ba1b5f4d78869a29019cdc7853 P 3793e3bed5e04acb8adbc620baf660a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:09:45.119237 20533 ts_tablet_manager.cc:1428] T 19aa73ba1b5f4d78869a29019cdc7853 P bffb811d93894ea5a14e6f4f56f4a196: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260926 15:09:45.119340 20531 raft_consensus.cc:729] T 19aa73ba1b5f4d78869a29019cdc7853 P 3793e3bed5e04acb8adbc620baf660a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3793e3bed5e04acb8adbc620baf660a4, State: Initialized, Role: FOLLOWER
I20260926 15:09:45.119614 20531 consensus_queue.cc:260] T 19aa73ba1b5f4d78869a29019cdc7853 P 3793e3bed5e04acb8adbc620baf660a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bffb811d93894ea5a14e6f4f56f4a196" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 44487 } } peers { permanent_uuid: "641358d46138484794ffdf3e0ceb4109" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 35581 } } peers { permanent_uuid: "3793e3bed5e04acb8adbc620baf660a4" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 45155 } }
I20260926 15:09:45.120226 20531 ts_tablet_manager.cc:1428] T 19aa73ba1b5f4d78869a29019cdc7853 P 3793e3bed5e04acb8adbc620baf660a4: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260926 15:09:45.308712 20538 raft_consensus.cc:482] T 19aa73ba1b5f4d78869a29019cdc7853 P 641358d46138484794ffdf3e0ceb4109 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 15:09:45.308923 20538 raft_consensus.cc:504] T 19aa73ba1b5f4d78869a29019cdc7853 P 641358d46138484794ffdf3e0ceb4109 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bffb811d93894ea5a14e6f4f56f4a196" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 44487 } } peers { permanent_uuid: "641358d46138484794ffdf3e0ceb4109" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 35581 } } peers { permanent_uuid: "3793e3bed5e04acb8adbc620baf660a4" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 45155 } }
I20260926 15:09:45.310400 20538 leader_election.cc:288] T 19aa73ba1b5f4d78869a29019cdc7853 P 641358d46138484794ffdf3e0ceb4109 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bffb811d93894ea5a14e6f4f56f4a196 (127.12.34.193:44487), 3793e3bed5e04acb8adbc620baf660a4 (127.12.34.195:45155)
I20260926 15:09:45.317832 20614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:45.310622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.195:45155 (local address 127.12.34.194:44935)
0926 15:09:45.311164 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0926 15:09:45.311188 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 15:09:45.311398 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:09:45.311723 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:09:45.311728 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:09:45.311751 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:09:45.312094 (+ 343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:45.312105 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:45.313266 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:45.313274 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:45.315900 (+ 2626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:45.315914 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:45.317156 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:45.317165 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:45.317342 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:45.317367 (+ 25us) client_negotiation.cc:769] Sending connection context
0926 15:09:45.317455 (+ 88us) client_negotiation.cc:240] Negotiation successful
0926 15:09:45.317546 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":374,"mutex_wait_us":110,"thread_start_us":240,"threads_started":1}
I20260926 15:09:45.317917 20526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:45.310744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.194:44935 (local address 127.12.34.195:45155)
0926 15:09:45.310987 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:45.310993 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:45.311281 (+ 288us) server_negotiation.cc:407] Connection header received
0926 15:09:45.311544 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:45.311555 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:45.311614 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:45.311710 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:09:45.312252 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:45.313062 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:45.316061 (+ 2999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:45.316882 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:45.317024 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:45.317133 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 15:09:45.317499 (+ 366us) server_negotiation.cc:299] Negotiation successful
0926 15:09:45.317638 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260926 15:09:45.318552 20461 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19aa73ba1b5f4d78869a29019cdc7853" candidate_uuid: "641358d46138484794ffdf3e0ceb4109" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3793e3bed5e04acb8adbc620baf660a4" is_pre_election: true
I20260926 15:09:45.318830 20461 raft_consensus.cc:2454] T 19aa73ba1b5f4d78869a29019cdc7853 P 3793e3bed5e04acb8adbc620baf660a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 641358d46138484794ffdf3e0ceb4109 in term 0.
I20260926 15:09:45.319378 20581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:45.311158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.194:34157 (local address 127.12.34.193:44487)
0926 15:09:45.311346 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:45.311351 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:45.311577 (+ 226us) server_negotiation.cc:407] Connection header received
0926 15:09:45.311772 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:45.311776 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:45.311840 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:45.311941 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:09:45.312507 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:45.313299 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:45.317206 (+ 3907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:45.318380 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:45.318503 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:45.318551 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0926 15:09:45.319051 (+ 500us) server_negotiation.cc:299] Negotiation successful
0926 15:09:45.319193 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 15:09:45.319285 20248 leader_election.cc:302] T 19aa73ba1b5f4d78869a29019cdc7853 P 641358d46138484794ffdf3e0ceb4109 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3793e3bed5e04acb8adbc620baf660a4, 641358d46138484794ffdf3e0ceb4109; no voters:
I20260926 15:09:45.319451 20615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:45.310844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.193:44487 (local address 127.12.34.194:34157)
0926 15:09:45.311477 (+ 633us) negotiation.cc:107] Waiting for socket to connect
0926 15:09:45.311495 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 15:09:45.311631 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:09:45.311953 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:09:45.311958 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:09:45.311979 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:09:45.312357 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:45.312365 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:45.313432 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:45.313437 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:45.315906 (+ 2469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:45.315914 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:45.318594 (+ 2680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:45.318600 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:45.318801 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:45.318843 (+ 42us) client_negotiation.cc:769] Sending connection context
0926 15:09:45.318942 (+ 99us) client_negotiation.cc:240] Negotiation successful
0926 15:09:45.319063 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"mutex_wait_us":128,"thread_start_us":307,"threads_started":1}
I20260926 15:09:45.319715 20538 raft_consensus.cc:2781] T 19aa73ba1b5f4d78869a29019cdc7853 P 641358d46138484794ffdf3e0ceb4109 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 15:09:45.319868 20538 raft_consensus.cc:482] T 19aa73ba1b5f4d78869a29019cdc7853 P 641358d46138484794ffdf3e0ceb4109 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 15:09:45.319971 20538 raft_consensus.cc:3037] T 19aa73ba1b5f4d78869a29019cdc7853 P 641358d46138484794ffdf3e0ceb4109 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:09:45.322110 20538 raft_consensus.cc:504] T 19aa73ba1b5f4d78869a29019cdc7853 P 641358d46138484794ffdf3e0ceb4109 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bffb811d93894ea5a14e6f4f56f4a196" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 44487 } } peers { permanent_uuid: "641358d46138484794ffdf3e0ceb4109" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 35581 } } peers { permanent_uuid: "3793e3bed5e04acb8adbc620baf660a4" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 45155 } }
I20260926 15:09:45.322773 20538 leader_election.cc:288] T 19aa73ba1b5f4d78869a29019cdc7853 P 641358d46138484794ffdf3e0ceb4109 [CANDIDATE]: Term 1 election: Requested vote from peers bffb811d93894ea5a14e6f4f56f4a196 (127.12.34.193:44487), 3793e3bed5e04acb8adbc620baf660a4 (127.12.34.195:45155)
I20260926 15:09:45.322778 20178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19aa73ba1b5f4d78869a29019cdc7853" candidate_uuid: "641358d46138484794ffdf3e0ceb4109" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bffb811d93894ea5a14e6f4f56f4a196" is_pre_election: true
I20260926 15:09:45.323069 20178 raft_consensus.cc:2454] T 19aa73ba1b5f4d78869a29019cdc7853 P bffb811d93894ea5a14e6f4f56f4a196 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 641358d46138484794ffdf3e0ceb4109 in term 0.
I20260926 15:09:45.323287 20461 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19aa73ba1b5f4d78869a29019cdc7853" candidate_uuid: "641358d46138484794ffdf3e0ceb4109" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3793e3bed5e04acb8adbc620baf660a4"
I20260926 15:09:45.323254 20177 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19aa73ba1b5f4d78869a29019cdc7853" candidate_uuid: "641358d46138484794ffdf3e0ceb4109" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bffb811d93894ea5a14e6f4f56f4a196"
I20260926 15:09:45.323479 20461 raft_consensus.cc:3037] T 19aa73ba1b5f4d78869a29019cdc7853 P 3793e3bed5e04acb8adbc620baf660a4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:09:45.323479 20177 raft_consensus.cc:3037] T 19aa73ba1b5f4d78869a29019cdc7853 P bffb811d93894ea5a14e6f4f56f4a196 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:09:45.326364 20177 raft_consensus.cc:2454] T 19aa73ba1b5f4d78869a29019cdc7853 P bffb811d93894ea5a14e6f4f56f4a196 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 641358d46138484794ffdf3e0ceb4109 in term 1.
I20260926 15:09:45.326370 20461 raft_consensus.cc:2454] T 19aa73ba1b5f4d78869a29019cdc7853 P 3793e3bed5e04acb8adbc620baf660a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 641358d46138484794ffdf3e0ceb4109 in term 1.
I20260926 15:09:45.326762 20251 leader_election.cc:302] T 19aa73ba1b5f4d78869a29019cdc7853 P 641358d46138484794ffdf3e0ceb4109 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 641358d46138484794ffdf3e0ceb4109, bffb811d93894ea5a14e6f4f56f4a196; no voters:
I20260926 15:09:45.327068 20538 raft_consensus.cc:2781] T 19aa73ba1b5f4d78869a29019cdc7853 P 641358d46138484794ffdf3e0ceb4109 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:09:45.327456 20538 raft_consensus.cc:686] T 19aa73ba1b5f4d78869a29019cdc7853 P 641358d46138484794ffdf3e0ceb4109 [term 1 LEADER]: Becoming Leader. State: Replica: 641358d46138484794ffdf3e0ceb4109, State: Running, Role: LEADER
I20260926 15:09:45.327808 20538 consensus_queue.cc:237] T 19aa73ba1b5f4d78869a29019cdc7853 P 641358d46138484794ffdf3e0ceb4109 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bffb811d93894ea5a14e6f4f56f4a196" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 44487 } } peers { permanent_uuid: "641358d46138484794ffdf3e0ceb4109" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 35581 } } peers { permanent_uuid: "3793e3bed5e04acb8adbc620baf660a4" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 45155 } }
I20260926 15:09:45.331175 20021 catalog_manager.cc:5495] T 19aa73ba1b5f4d78869a29019cdc7853 P 641358d46138484794ffdf3e0ceb4109 reported cstate change: term changed from 0 to 1, leader changed from <none> to 641358d46138484794ffdf3e0ceb4109 (127.12.34.194). New cstate: current_term: 1 leader_uuid: "641358d46138484794ffdf3e0ceb4109" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bffb811d93894ea5a14e6f4f56f4a196" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 44487 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "641358d46138484794ffdf3e0ceb4109" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 35581 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3793e3bed5e04acb8adbc620baf660a4" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 45155 } health_report { overall_health: UNKNOWN } } }
I20260926 15:09:45.353466 20619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:45.348105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41174 (local address 127.12.34.194:35581)
0926 15:09:45.348577 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:45.348581 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:45.348596 (+ 15us) server_negotiation.cc:407] Connection header received
0926 15:09:45.348670 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:45.348673 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:45.348747 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:45.348883 (+ 136us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 15:09:45.349351 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:45.350066 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:45.351368 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:45.351813 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:45.351909 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:45.352399 (+ 490us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 15:09:45.352933 (+ 534us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 15:09:45.353038 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 15:09:45.353204 (+ 166us) server_negotiation.cc:299] Negotiation successful
0926 15:09:45.353266 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":280,"threads_started":1}
I20260926 15:09:45.356722 20461 raft_consensus.cc:1261] T 19aa73ba1b5f4d78869a29019cdc7853 P 3793e3bed5e04acb8adbc620baf660a4 [term 1 FOLLOWER]: Refusing update from remote peer 641358d46138484794ffdf3e0ceb4109: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:09:45.356779 20177 raft_consensus.cc:1261] T 19aa73ba1b5f4d78869a29019cdc7853 P bffb811d93894ea5a14e6f4f56f4a196 [term 1 FOLLOWER]: Refusing update from remote peer 641358d46138484794ffdf3e0ceb4109: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:09:45.357388 20616 consensus_queue.cc:1035] T 19aa73ba1b5f4d78869a29019cdc7853 P 641358d46138484794ffdf3e0ceb4109 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3793e3bed5e04acb8adbc620baf660a4" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 45155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:09:45.357724 20538 consensus_queue.cc:1035] T 19aa73ba1b5f4d78869a29019cdc7853 P 641358d46138484794ffdf3e0ceb4109 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bffb811d93894ea5a14e6f4f56f4a196" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 44487 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:09:45.373646 20020 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43092:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 15:09:45.373910 20020 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43092:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 15:09:45.377296 20020 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1a3d30fc883a4f67bc16c22629712a96: Sending DeleteTablet for 3 replicas of tablet 19aa73ba1b5f4d78869a29019cdc7853
I20260926 15:09:45.378084 20440 tablet_service.cc:1514] Processing DeleteTablet for tablet 19aa73ba1b5f4d78869a29019cdc7853 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:09:45 UTC) from {username='oryx', principal='oryx/127.12.34.254@KRBTEST.COM'} at 127.0.0.1:37212
I20260926 15:09:45.378088 20158 tablet_service.cc:1514] Processing DeleteTablet for tablet 19aa73ba1b5f4d78869a29019cdc7853 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:09:45 UTC) from {username='oryx', principal='oryx/127.12.34.254@KRBTEST.COM'} at 127.0.0.1:44032
I20260926 15:09:45.378427 20598 tablet_replica.cc:330] T 19aa73ba1b5f4d78869a29019cdc7853 P 3793e3bed5e04acb8adbc620baf660a4: stopping tablet replica
I20260926 15:09:45.378684 20598 raft_consensus.cc:2229] T 19aa73ba1b5f4d78869a29019cdc7853 P 3793e3bed5e04acb8adbc620baf660a4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 15:09:45.378871 20598 pending_rounds.cc:62] T 19aa73ba1b5f4d78869a29019cdc7853 P 3793e3bed5e04acb8adbc620baf660a4: Trying to abort 2 pending ops.
I20260926 15:09:45.378991 20598 pending_rounds.cc:68] T 19aa73ba1b5f4d78869a29019cdc7853 P 3793e3bed5e04acb8adbc620baf660a4: Aborting op as it isn't in flight: id { term: 1 index: 2 } timestamp: 7333623338415271936 op_type: WRITE_OP write_request { tablet_id: "19aa73ba1b5f4d78869a29019cdc7853" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "val" type: INT32 is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\003\00090\000\0001\324\000\000" indirect_data: "<redacted>""" } external_consistency_mode: CLIENT_PROPAGATED authz_token { token_data: "\010\345\302\337\325\006\"8\n\ntest-admin\022*\n ac99adce1e7e438498ecc0d7da93c5ff\020\001\030\001 \001(\001" signature: "<redacted>""\244\234u\323\320\322\252 \375K\2424\206\235Tm\'\244\000\030\201p\2469-A\2427t8\003\237\315nb`f\335\200\334\021\332J\237M\351y\213\231\251\227K.\003}\027E2\226\3023DZu" signing_key_seq_num: 0 } } request_id { client_id: "fca67b44b2a24d66a030b59f1bbb3ba7" seq_no: 0 first_incomplete_seq_no: 0 attempt_no: 0 }
I20260926 15:09:45.379779 20598 pending_rounds.cc:68] T 19aa73ba1b5f4d78869a29019cdc7853 P 3793e3bed5e04acb8adbc620baf660a4: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7333623338306523136 op_type: NO_OP noop_request { }
I20260926 15:09:45.379945 20598 raft_consensus.cc:2866] T 19aa73ba1b5f4d78869a29019cdc7853 P 3793e3bed5e04acb8adbc620baf660a4 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260926 15:09:45.380069 20598 raft_consensus.cc:2258] T 19aa73ba1b5f4d78869a29019cdc7853 P 3793e3bed5e04acb8adbc620baf660a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:09:45.380385 20596 tablet_replica.cc:330] T 19aa73ba1b5f4d78869a29019cdc7853 P bffb811d93894ea5a14e6f4f56f4a196: stopping tablet replica
W20260926 15:09:45.380491 20248 consensus_peers.cc:487] T 19aa73ba1b5f4d78869a29019cdc7853 P 641358d46138484794ffdf3e0ceb4109 -> Peer 3793e3bed5e04acb8adbc620baf660a4 (127.12.34.195:45155): Couldn't send request to peer 3793e3bed5e04acb8adbc620baf660a4. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: STOPPING. This is attempt 1: this message will repeat every 5th retry.
I20260926 15:09:45.380626 20596 raft_consensus.cc:2229] T 19aa73ba1b5f4d78869a29019cdc7853 P bffb811d93894ea5a14e6f4f56f4a196 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 15:09:45.380883 20596 raft_consensus.cc:2258] T 19aa73ba1b5f4d78869a29019cdc7853 P bffb811d93894ea5a14e6f4f56f4a196 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:09:45.382440 20296 tablet_service.cc:1514] Processing DeleteTablet for tablet 19aa73ba1b5f4d78869a29019cdc7853 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:09:45 UTC) from {username='oryx', principal='oryx/127.12.34.254@KRBTEST.COM'} at 127.0.0.1:41168
I20260926 15:09:45.383068 20597 tablet_replica.cc:330] T 19aa73ba1b5f4d78869a29019cdc7853 P 641358d46138484794ffdf3e0ceb4109: stopping tablet replica
I20260926 15:09:45.383332 20597 raft_consensus.cc:2229] T 19aa73ba1b5f4d78869a29019cdc7853 P 641358d46138484794ffdf3e0ceb4109 [term 1 LEADER]: Raft consensus shutting down.
I20260926 15:09:45.384811 20596 ts_tablet_manager.cc:1905] T 19aa73ba1b5f4d78869a29019cdc7853 P bffb811d93894ea5a14e6f4f56f4a196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:09:45.386783 20597 raft_consensus.cc:2258] T 19aa73ba1b5f4d78869a29019cdc7853 P 641358d46138484794ffdf3e0ceb4109 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:09:45.387686 20020 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43092:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 15:09:45.387899 20597 ts_tablet_manager.cc:1905] T 19aa73ba1b5f4d78869a29019cdc7853 P 641358d46138484794ffdf3e0ceb4109: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260926 15:09:45.388334 20020 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 15:09:45.391539 20596 ts_tablet_manager.cc:1918] T 19aa73ba1b5f4d78869a29019cdc7853 P bffb811d93894ea5a14e6f4f56f4a196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 15:09:45.391688 20596 log.cc:1196] T 19aa73ba1b5f4d78869a29019cdc7853 P bffb811d93894ea5a14e6f4f56f4a196: Deleting WAL directory at /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435318278681-12427-0/minicluster-data/ts-0/wal/wals/19aa73ba1b5f4d78869a29019cdc7853
I20260926 15:09:45.392153 20596 ts_tablet_manager.cc:1939] T 19aa73ba1b5f4d78869a29019cdc7853 P bffb811d93894ea5a14e6f4f56f4a196: Deleting consensus metadata
I20260926 15:09:45.393217 20008 catalog_manager.cc:4841] TS bffb811d93894ea5a14e6f4f56f4a196 (127.12.34.193:44487): tablet 19aa73ba1b5f4d78869a29019cdc7853 (table test-table [id=ac99adce1e7e438498ecc0d7da93c5ff]) successfully deleted
I20260926 15:09:45.394796 20597 ts_tablet_manager.cc:1918] T 19aa73ba1b5f4d78869a29019cdc7853 P 641358d46138484794ffdf3e0ceb4109: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 15:09:45.394942 20597 log.cc:1196] T 19aa73ba1b5f4d78869a29019cdc7853 P 641358d46138484794ffdf3e0ceb4109: Deleting WAL directory at /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435318278681-12427-0/minicluster-data/ts-1/wal/wals/19aa73ba1b5f4d78869a29019cdc7853
I20260926 15:09:45.395380 20597 ts_tablet_manager.cc:1939] T 19aa73ba1b5f4d78869a29019cdc7853 P 641358d46138484794ffdf3e0ceb4109: Deleting consensus metadata
I20260926 15:09:45.395921 20598 ts_tablet_manager.cc:1905] T 19aa73ba1b5f4d78869a29019cdc7853 P 3793e3bed5e04acb8adbc620baf660a4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:09:45.396004 20440 tablet_service.cc:1467] Processing CreateTablet for tablet 82cbed0f080540abb52490897028e6dd (DEFAULT_TABLE table=test-table [id=6f78c85d8b214cbf9e565d3a75a77c84]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:09:45.396394 20440 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82cbed0f080540abb52490897028e6dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:09:45.400069 20158 tablet_service.cc:1467] Processing CreateTablet for tablet 82cbed0f080540abb52490897028e6dd (DEFAULT_TABLE table=test-table [id=6f78c85d8b214cbf9e565d3a75a77c84]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:09:45.400332 20006 catalog_manager.cc:4841] TS 641358d46138484794ffdf3e0ceb4109 (127.12.34.194:35581): tablet 19aa73ba1b5f4d78869a29019cdc7853 (table test-table [id=ac99adce1e7e438498ecc0d7da93c5ff]) successfully deleted
I20260926 15:09:45.400476 20158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82cbed0f080540abb52490897028e6dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:09:45.401005 20296 tablet_service.cc:1467] Processing CreateTablet for tablet 82cbed0f080540abb52490897028e6dd (DEFAULT_TABLE table=test-table [id=6f78c85d8b214cbf9e565d3a75a77c84]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:09:45.401387 20296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82cbed0f080540abb52490897028e6dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:09:45.403684 20598 ts_tablet_manager.cc:1918] T 19aa73ba1b5f4d78869a29019cdc7853 P 3793e3bed5e04acb8adbc620baf660a4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 15:09:45.403893 20598 log.cc:1196] T 19aa73ba1b5f4d78869a29019cdc7853 P 3793e3bed5e04acb8adbc620baf660a4: Deleting WAL directory at /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435318278681-12427-0/minicluster-data/ts-2/wal/wals/19aa73ba1b5f4d78869a29019cdc7853
I20260926 15:09:45.404335 20598 ts_tablet_manager.cc:1939] T 19aa73ba1b5f4d78869a29019cdc7853 P 3793e3bed5e04acb8adbc620baf660a4: Deleting consensus metadata
I20260926 15:09:45.405644 20005 catalog_manager.cc:4841] TS 3793e3bed5e04acb8adbc620baf660a4 (127.12.34.195:45155): tablet 19aa73ba1b5f4d78869a29019cdc7853 (table test-table [id=ac99adce1e7e438498ecc0d7da93c5ff]) successfully deleted
I20260926 15:09:45.407688 20533 tablet_bootstrap.cc:492] T 82cbed0f080540abb52490897028e6dd P bffb811d93894ea5a14e6f4f56f4a196: Bootstrap starting.
I20260926 15:09:45.408016 20531 tablet_bootstrap.cc:492] T 82cbed0f080540abb52490897028e6dd P 3793e3bed5e04acb8adbc620baf660a4: Bootstrap starting.
I20260926 15:09:45.409334 20533 tablet_bootstrap.cc:654] T 82cbed0f080540abb52490897028e6dd P bffb811d93894ea5a14e6f4f56f4a196: Neither blocks nor log segments found. Creating new log.
I20260926 15:09:45.409578 20531 tablet_bootstrap.cc:654] T 82cbed0f080540abb52490897028e6dd P 3793e3bed5e04acb8adbc620baf660a4: Neither blocks nor log segments found. Creating new log.
I20260926 15:09:45.410104 20532 tablet_bootstrap.cc:492] T 82cbed0f080540abb52490897028e6dd P 641358d46138484794ffdf3e0ceb4109: Bootstrap starting.
I20260926 15:09:45.411701 20532 tablet_bootstrap.cc:654] T 82cbed0f080540abb52490897028e6dd P 641358d46138484794ffdf3e0ceb4109: Neither blocks nor log segments found. Creating new log.
I20260926 15:09:45.420375 20533 tablet_bootstrap.cc:492] T 82cbed0f080540abb52490897028e6dd P bffb811d93894ea5a14e6f4f56f4a196: No bootstrap required, opened a new log
I20260926 15:09:45.420487 20531 tablet_bootstrap.cc:492] T 82cbed0f080540abb52490897028e6dd P 3793e3bed5e04acb8adbc620baf660a4: No bootstrap required, opened a new log
I20260926 15:09:45.420532 20533 ts_tablet_manager.cc:1397] T 82cbed0f080540abb52490897028e6dd P bffb811d93894ea5a14e6f4f56f4a196: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.003s
I20260926 15:09:45.420622 20531 ts_tablet_manager.cc:1397] T 82cbed0f080540abb52490897028e6dd P 3793e3bed5e04acb8adbc620baf660a4: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.003s
I20260926 15:09:45.421259 20533 raft_consensus.cc:348] T 82cbed0f080540abb52490897028e6dd P bffb811d93894ea5a14e6f4f56f4a196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bffb811d93894ea5a14e6f4f56f4a196" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 44487 } } peers { permanent_uuid: "641358d46138484794ffdf3e0ceb4109" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 35581 } } peers { permanent_uuid: "3793e3bed5e04acb8adbc620baf660a4" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 45155 } }
I20260926 15:09:45.421330 20531 raft_consensus.cc:348] T 82cbed0f080540abb52490897028e6dd P 3793e3bed5e04acb8adbc620baf660a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bffb811d93894ea5a14e6f4f56f4a196" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 44487 } } peers { permanent_uuid: "641358d46138484794ffdf3e0ceb4109" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 35581 } } peers { permanent_uuid: "3793e3bed5e04acb8adbc620baf660a4" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 45155 } }
I20260926 15:09:45.421525 20533 raft_consensus.cc:374] T 82cbed0f080540abb52490897028e6dd P bffb811d93894ea5a14e6f4f56f4a196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:09:45.421540 20531 raft_consensus.cc:374] T 82cbed0f080540abb52490897028e6dd P 3793e3bed5e04acb8adbc620baf660a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:09:45.421607 20533 raft_consensus.cc:729] T 82cbed0f080540abb52490897028e6dd P bffb811d93894ea5a14e6f4f56f4a196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bffb811d93894ea5a14e6f4f56f4a196, State: Initialized, Role: FOLLOWER
I20260926 15:09:45.421607 20531 raft_consensus.cc:729] T 82cbed0f080540abb52490897028e6dd P 3793e3bed5e04acb8adbc620baf660a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3793e3bed5e04acb8adbc620baf660a4, State: Initialized, Role: FOLLOWER
I20260926 15:09:45.421864 20531 consensus_queue.cc:260] T 82cbed0f080540abb52490897028e6dd P 3793e3bed5e04acb8adbc620baf660a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bffb811d93894ea5a14e6f4f56f4a196" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 44487 } } peers { permanent_uuid: "641358d46138484794ffdf3e0ceb4109" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 35581 } } peers { permanent_uuid: "3793e3bed5e04acb8adbc620baf660a4" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 45155 } }
I20260926 15:09:45.421854 20533 consensus_queue.cc:260] T 82cbed0f080540abb52490897028e6dd P bffb811d93894ea5a14e6f4f56f4a196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bffb811d93894ea5a14e6f4f56f4a196" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 44487 } } peers { permanent_uuid: "641358d46138484794ffdf3e0ceb4109" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 35581 } } peers { permanent_uuid: "3793e3bed5e04acb8adbc620baf660a4" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 45155 } }
I20260926 15:09:45.422472 20533 ts_tablet_manager.cc:1428] T 82cbed0f080540abb52490897028e6dd P bffb811d93894ea5a14e6f4f56f4a196: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 15:09:45.422892 20531 ts_tablet_manager.cc:1428] T 82cbed0f080540abb52490897028e6dd P 3793e3bed5e04acb8adbc620baf660a4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 15:09:45.427138 20532 tablet_bootstrap.cc:492] T 82cbed0f080540abb52490897028e6dd P 641358d46138484794ffdf3e0ceb4109: No bootstrap required, opened a new log
I20260926 15:09:45.427318 20532 ts_tablet_manager.cc:1397] T 82cbed0f080540abb52490897028e6dd P 641358d46138484794ffdf3e0ceb4109: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.008s
I20260926 15:09:45.428095 20532 raft_consensus.cc:348] T 82cbed0f080540abb52490897028e6dd P 641358d46138484794ffdf3e0ceb4109 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bffb811d93894ea5a14e6f4f56f4a196" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 44487 } } peers { permanent_uuid: "641358d46138484794ffdf3e0ceb4109" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 35581 } } peers { permanent_uuid: "3793e3bed5e04acb8adbc620baf660a4" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 45155 } }
I20260926 15:09:45.428395 20532 raft_consensus.cc:374] T 82cbed0f080540abb52490897028e6dd P 641358d46138484794ffdf3e0ceb4109 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:09:45.428485 20532 raft_consensus.cc:729] T 82cbed0f080540abb52490897028e6dd P 641358d46138484794ffdf3e0ceb4109 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 641358d46138484794ffdf3e0ceb4109, State: Initialized, Role: FOLLOWER
I20260926 15:09:45.428740 20532 consensus_queue.cc:260] T 82cbed0f080540abb52490897028e6dd P 641358d46138484794ffdf3e0ceb4109 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bffb811d93894ea5a14e6f4f56f4a196" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 44487 } } peers { permanent_uuid: "641358d46138484794ffdf3e0ceb4109" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 35581 } } peers { permanent_uuid: "3793e3bed5e04acb8adbc620baf660a4" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 45155 } }
I20260926 15:09:45.429265 20532 ts_tablet_manager.cc:1428] T 82cbed0f080540abb52490897028e6dd P 641358d46138484794ffdf3e0ceb4109: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260926 15:09:45.825976 20537 raft_consensus.cc:482] T 82cbed0f080540abb52490897028e6dd P 3793e3bed5e04acb8adbc620baf660a4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 15:09:45.826233 20537 raft_consensus.cc:504] T 82cbed0f080540abb52490897028e6dd P 3793e3bed5e04acb8adbc620baf660a4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bffb811d93894ea5a14e6f4f56f4a196" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 44487 } } peers { permanent_uuid: "641358d46138484794ffdf3e0ceb4109" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 35581 } } peers { permanent_uuid: "3793e3bed5e04acb8adbc620baf660a4" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 45155 } }
I20260926 15:09:45.826861 20537 leader_election.cc:288] T 82cbed0f080540abb52490897028e6dd P 3793e3bed5e04acb8adbc620baf660a4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bffb811d93894ea5a14e6f4f56f4a196 (127.12.34.193:44487), 641358d46138484794ffdf3e0ceb4109 (127.12.34.194:35581)
I20260926 15:09:45.827365 20177 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82cbed0f080540abb52490897028e6dd" candidate_uuid: "3793e3bed5e04acb8adbc620baf660a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bffb811d93894ea5a14e6f4f56f4a196" is_pre_election: true
I20260926 15:09:45.827399 20322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82cbed0f080540abb52490897028e6dd" candidate_uuid: "3793e3bed5e04acb8adbc620baf660a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "641358d46138484794ffdf3e0ceb4109" is_pre_election: true
I20260926 15:09:45.827589 20177 raft_consensus.cc:2454] T 82cbed0f080540abb52490897028e6dd P bffb811d93894ea5a14e6f4f56f4a196 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3793e3bed5e04acb8adbc620baf660a4 in term 0.
I20260926 15:09:45.827642 20322 raft_consensus.cc:2454] T 82cbed0f080540abb52490897028e6dd P 641358d46138484794ffdf3e0ceb4109 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3793e3bed5e04acb8adbc620baf660a4 in term 0.
I20260926 15:09:45.828027 20390 leader_election.cc:302] T 82cbed0f080540abb52490897028e6dd P 3793e3bed5e04acb8adbc620baf660a4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3793e3bed5e04acb8adbc620baf660a4, bffb811d93894ea5a14e6f4f56f4a196; no voters:
I20260926 15:09:45.828329 20537 raft_consensus.cc:2781] T 82cbed0f080540abb52490897028e6dd P 3793e3bed5e04acb8adbc620baf660a4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 15:09:45.828455 20537 raft_consensus.cc:482] T 82cbed0f080540abb52490897028e6dd P 3793e3bed5e04acb8adbc620baf660a4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 15:09:45.828567 20537 raft_consensus.cc:3037] T 82cbed0f080540abb52490897028e6dd P 3793e3bed5e04acb8adbc620baf660a4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:09:45.830623 20537 raft_consensus.cc:504] T 82cbed0f080540abb52490897028e6dd P 3793e3bed5e04acb8adbc620baf660a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bffb811d93894ea5a14e6f4f56f4a196" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 44487 } } peers { permanent_uuid: "641358d46138484794ffdf3e0ceb4109" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 35581 } } peers { permanent_uuid: "3793e3bed5e04acb8adbc620baf660a4" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 45155 } }
I20260926 15:09:45.831149 20537 leader_election.cc:288] T 82cbed0f080540abb52490897028e6dd P 3793e3bed5e04acb8adbc620baf660a4 [CANDIDATE]: Term 1 election: Requested vote from peers bffb811d93894ea5a14e6f4f56f4a196 (127.12.34.193:44487), 641358d46138484794ffdf3e0ceb4109 (127.12.34.194:35581)
I20260926 15:09:45.831550 20177 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82cbed0f080540abb52490897028e6dd" candidate_uuid: "3793e3bed5e04acb8adbc620baf660a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bffb811d93894ea5a14e6f4f56f4a196"
I20260926 15:09:45.831552 20322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82cbed0f080540abb52490897028e6dd" candidate_uuid: "3793e3bed5e04acb8adbc620baf660a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "641358d46138484794ffdf3e0ceb4109"
I20260926 15:09:45.831748 20322 raft_consensus.cc:3037] T 82cbed0f080540abb52490897028e6dd P 641358d46138484794ffdf3e0ceb4109 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:09:45.831741 20177 raft_consensus.cc:3037] T 82cbed0f080540abb52490897028e6dd P bffb811d93894ea5a14e6f4f56f4a196 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:09:45.833699 20177 raft_consensus.cc:2454] T 82cbed0f080540abb52490897028e6dd P bffb811d93894ea5a14e6f4f56f4a196 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3793e3bed5e04acb8adbc620baf660a4 in term 1.
I20260926 15:09:45.834080 20390 leader_election.cc:302] T 82cbed0f080540abb52490897028e6dd P 3793e3bed5e04acb8adbc620baf660a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3793e3bed5e04acb8adbc620baf660a4, bffb811d93894ea5a14e6f4f56f4a196; no voters:
I20260926 15:09:45.834336 20537 raft_consensus.cc:2781] T 82cbed0f080540abb52490897028e6dd P 3793e3bed5e04acb8adbc620baf660a4 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:09:45.834503 20322 raft_consensus.cc:2454] T 82cbed0f080540abb52490897028e6dd P 641358d46138484794ffdf3e0ceb4109 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3793e3bed5e04acb8adbc620baf660a4 in term 1.
I20260926 15:09:45.834803 20537 raft_consensus.cc:686] T 82cbed0f080540abb52490897028e6dd P 3793e3bed5e04acb8adbc620baf660a4 [term 1 LEADER]: Becoming Leader. State: Replica: 3793e3bed5e04acb8adbc620baf660a4, State: Running, Role: LEADER
I20260926 15:09:45.835089 20537 consensus_queue.cc:237] T 82cbed0f080540abb52490897028e6dd P 3793e3bed5e04acb8adbc620baf660a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bffb811d93894ea5a14e6f4f56f4a196" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 44487 } } peers { permanent_uuid: "641358d46138484794ffdf3e0ceb4109" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 35581 } } peers { permanent_uuid: "3793e3bed5e04acb8adbc620baf660a4" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 45155 } }
I20260926 15:09:45.837432 20020 catalog_manager.cc:5495] T 82cbed0f080540abb52490897028e6dd P 3793e3bed5e04acb8adbc620baf660a4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3793e3bed5e04acb8adbc620baf660a4 (127.12.34.195). New cstate: current_term: 1 leader_uuid: "3793e3bed5e04acb8adbc620baf660a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bffb811d93894ea5a14e6f4f56f4a196" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 44487 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "641358d46138484794ffdf3e0ceb4109" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 35581 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3793e3bed5e04acb8adbc620baf660a4" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 45155 } health_report { overall_health: HEALTHY } } }
I20260926 15:09:45.871739 20636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:45.867204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37232 (local address 127.12.34.195:45155)
0926 15:09:45.867614 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:45.867618 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:45.867636 (+ 18us) server_negotiation.cc:407] Connection header received
0926 15:09:45.867803 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:45.867806 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:45.867866 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:45.867945 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 15:09:45.868303 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:45.869048 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:45.870318 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:45.870743 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:45.870848 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:45.871177 (+ 329us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 15:09:45.871272 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 15:09:45.871350 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 15:09:45.871440 (+ 90us) server_negotiation.cc:299] Negotiation successful
0926 15:09:45.871494 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":238,"threads_started":1}
I20260926 15:09:45.879540 20322 raft_consensus.cc:1261] T 82cbed0f080540abb52490897028e6dd P 641358d46138484794ffdf3e0ceb4109 [term 1 FOLLOWER]: Refusing update from remote peer 3793e3bed5e04acb8adbc620baf660a4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:09:45.879705 20177 raft_consensus.cc:1261] T 82cbed0f080540abb52490897028e6dd P bffb811d93894ea5a14e6f4f56f4a196 [term 1 FOLLOWER]: Refusing update from remote peer 3793e3bed5e04acb8adbc620baf660a4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:09:45.880077 20627 consensus_queue.cc:1035] T 82cbed0f080540abb52490897028e6dd P 3793e3bed5e04acb8adbc620baf660a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "641358d46138484794ffdf3e0ceb4109" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 35581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:09:45.880422 20537 consensus_queue.cc:1035] T 82cbed0f080540abb52490897028e6dd P 3793e3bed5e04acb8adbc620baf660a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bffb811d93894ea5a14e6f4f56f4a196" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 44487 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:09:45.921785 20021 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43092:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 15:09:45.921988 20021 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43092:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 15:09:45.924620 20021 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1a3d30fc883a4f67bc16c22629712a96: Sending DeleteTablet for 3 replicas of tablet 82cbed0f080540abb52490897028e6dd
I20260926 15:09:45.925303 20158 tablet_service.cc:1514] Processing DeleteTablet for tablet 82cbed0f080540abb52490897028e6dd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:09:45 UTC) from {username='oryx', principal='oryx/127.12.34.254@KRBTEST.COM'} at 127.0.0.1:44032
I20260926 15:09:45.925474 20296 tablet_service.cc:1514] Processing DeleteTablet for tablet 82cbed0f080540abb52490897028e6dd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:09:45 UTC) from {username='oryx', principal='oryx/127.12.34.254@KRBTEST.COM'} at 127.0.0.1:41168
I20260926 15:09:45.925565 20440 tablet_service.cc:1514] Processing DeleteTablet for tablet 82cbed0f080540abb52490897028e6dd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:09:45 UTC) from {username='oryx', principal='oryx/127.12.34.254@KRBTEST.COM'} at 127.0.0.1:37212
I20260926 15:09:45.926095 20644 tablet_replica.cc:330] T 82cbed0f080540abb52490897028e6dd P 3793e3bed5e04acb8adbc620baf660a4: stopping tablet replica
I20260926 15:09:45.926280 20644 raft_consensus.cc:2229] T 82cbed0f080540abb52490897028e6dd P 3793e3bed5e04acb8adbc620baf660a4 [term 1 LEADER]: Raft consensus shutting down.
I20260926 15:09:45.926561 12427 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 20092
I20260926 15:09:45.926826 20644 raft_consensus.cc:2258] T 82cbed0f080540abb52490897028e6dd P 3793e3bed5e04acb8adbc620baf660a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:09:45.927313 20643 tablet_replica.cc:330] T 82cbed0f080540abb52490897028e6dd P 641358d46138484794ffdf3e0ceb4109: stopping tablet replica
I20260926 15:09:45.927516 20643 raft_consensus.cc:2229] T 82cbed0f080540abb52490897028e6dd P 641358d46138484794ffdf3e0ceb4109 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 15:09:45.927731 20643 raft_consensus.cc:2258] T 82cbed0f080540abb52490897028e6dd P 641358d46138484794ffdf3e0ceb4109 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:09:45.927788 20644 ts_tablet_manager.cc:1905] T 82cbed0f080540abb52490897028e6dd P 3793e3bed5e04acb8adbc620baf660a4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:09:45.928632 20643 ts_tablet_manager.cc:1905] T 82cbed0f080540abb52490897028e6dd P 641358d46138484794ffdf3e0ceb4109: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:09:45.936388 20643 ts_tablet_manager.cc:1918] T 82cbed0f080540abb52490897028e6dd P 641358d46138484794ffdf3e0ceb4109: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 15:09:45.936529 20643 log.cc:1196] T 82cbed0f080540abb52490897028e6dd P 641358d46138484794ffdf3e0ceb4109: Deleting WAL directory at /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435318278681-12427-0/minicluster-data/ts-1/wal/wals/82cbed0f080540abb52490897028e6dd
I20260926 15:09:45.936992 20643 ts_tablet_manager.cc:1939] T 82cbed0f080540abb52490897028e6dd P 641358d46138484794ffdf3e0ceb4109: Deleting consensus metadata
I20260926 15:09:45.937155 20644 ts_tablet_manager.cc:1918] T 82cbed0f080540abb52490897028e6dd P 3793e3bed5e04acb8adbc620baf660a4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 15:09:45.937305 20644 log.cc:1196] T 82cbed0f080540abb52490897028e6dd P 3793e3bed5e04acb8adbc620baf660a4: Deleting WAL directory at /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435318278681-12427-0/minicluster-data/ts-2/wal/wals/82cbed0f080540abb52490897028e6dd
I20260926 15:09:45.937668 20644 ts_tablet_manager.cc:1939] T 82cbed0f080540abb52490897028e6dd P 3793e3bed5e04acb8adbc620baf660a4: Deleting consensus metadata
I20260926 15:09:45.938540 20005 catalog_manager.cc:4841] TS 3793e3bed5e04acb8adbc620baf660a4 (127.12.34.195:45155): tablet 82cbed0f080540abb52490897028e6dd (table test-table [id=6f78c85d8b214cbf9e565d3a75a77c84]) successfully deleted
I20260926 15:09:45.938572 20006 catalog_manager.cc:4841] TS 641358d46138484794ffdf3e0ceb4109 (127.12.34.194:35581): tablet 82cbed0f080540abb52490897028e6dd (table test-table [id=6f78c85d8b214cbf9e565d3a75a77c84]) successfully deleted
I20260926 15:09:45.941942 20642 tablet_replica.cc:330] T 82cbed0f080540abb52490897028e6dd P bffb811d93894ea5a14e6f4f56f4a196: stopping tablet replica
I20260926 15:09:45.942148 20642 raft_consensus.cc:2229] T 82cbed0f080540abb52490897028e6dd P bffb811d93894ea5a14e6f4f56f4a196 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 15:09:45.942363 20642 raft_consensus.cc:2258] T 82cbed0f080540abb52490897028e6dd P bffb811d93894ea5a14e6f4f56f4a196 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:09:45.943116 20642 ts_tablet_manager.cc:1905] T 82cbed0f080540abb52490897028e6dd P bffb811d93894ea5a14e6f4f56f4a196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:09:45.947729 20642 ts_tablet_manager.cc:1918] T 82cbed0f080540abb52490897028e6dd P bffb811d93894ea5a14e6f4f56f4a196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 15:09:45.947868 20642 log.cc:1196] T 82cbed0f080540abb52490897028e6dd P bffb811d93894ea5a14e6f4f56f4a196: Deleting WAL directory at /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790435318278681-12427-0/minicluster-data/ts-0/wal/wals/82cbed0f080540abb52490897028e6dd
I20260926 15:09:45.948308 20642 ts_tablet_manager.cc:1939] T 82cbed0f080540abb52490897028e6dd P bffb811d93894ea5a14e6f4f56f4a196: Deleting consensus metadata
I20260926 15:09:45.949218 20008 catalog_manager.cc:4841] TS bffb811d93894ea5a14e6f4f56f4a196 (127.12.34.193:44487): tablet 82cbed0f080540abb52490897028e6dd (table test-table [id=6f78c85d8b214cbf9e565d3a75a77c84]) successfully deleted
Sep 26 15:09:45 dist-test-slave-865v krb5kdc[19976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435385, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.12.34.193@KRBTEST.COM
I20260926 15:09:45.955468 20646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:45.928585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44052 (local address 127.12.34.193:44487)
0926 15:09:45.937470 (+ 8885us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:45.937479 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:45.941586 (+ 4107us) server_negotiation.cc:407] Connection header received
0926 15:09:45.941747 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:45.941753 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:45.941831 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:45.941950 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:09:45.943284 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:45.944044 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:45.945309 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:45.945850 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:45.945980 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:45.950236 (+ 4256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:09:45.950249 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:09:45.950265 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:09:45.950298 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:09:45.952859 (+ 2561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:45.953521 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:45.953528 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:45.953530 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:45.953644 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:45.954012 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:45.954023 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:45.954026 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:45.954334 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:09:45.954637 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0926 15:09:45.954910 (+ 273us) server_negotiation.cc:299] Negotiation successful
0926 15:09:45.955126 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8718,"thread_start_us":8602,"threads_started":1}
I20260926 15:09:45.956045 20223 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:09:46.075116 12427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8uoJGv/build/asan/bin/kudu with pid 20092
I20260926 15:09:46.097483 12427 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 20232
Sep 26 15:09:46 dist-test-slave-865v krb5kdc[19976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435385, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.12.34.194@KRBTEST.COM
I20260926 15:09:46.112114 20648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:46.098224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41188 (local address 127.12.34.194:35581)
0926 15:09:46.098713 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:46.098717 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:46.098733 (+ 16us) server_negotiation.cc:407] Connection header received
0926 15:09:46.098863 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:46.098867 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:46.098934 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:46.099029 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:09:46.100415 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:46.101431 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:46.102637 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:46.103023 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:46.103123 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:46.107157 (+ 4034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:09:46.107169 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:09:46.107195 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:09:46.107250 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:09:46.110011 (+ 2761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:46.110592 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:46.110599 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:46.110601 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:46.110737 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:46.111019 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:46.111022 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:46.111024 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:46.111327 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:09:46.111550 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0926 15:09:46.111705 (+ 155us) server_negotiation.cc:299] Negotiation successful
0926 15:09:46.111855 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399,"thread_start_us":282,"threads_started":1}
I20260926 15:09:46.112766 20362 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:09:46.235168 12427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8uoJGv/build/asan/bin/kudu with pid 20232
I20260926 15:09:46.257175 12427 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 20371
Sep 26 15:09:46 dist-test-slave-865v krb5kdc[19976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435385, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.12.34.195@KRBTEST.COM
I20260926 15:09:46.272104 20636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:46.257866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37240 (local address 127.12.34.195:45155)
0926 15:09:46.258050 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:46.258057 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:46.258121 (+ 64us) server_negotiation.cc:407] Connection header received
0926 15:09:46.258359 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:46.258363 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:46.258446 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:46.258559 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:09:46.259803 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:46.260551 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:46.261962 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:46.262358 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:46.262487 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:46.266771 (+ 4284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:09:46.266783 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:09:46.266799 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:09:46.266832 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:09:46.269580 (+ 2748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:46.270220 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:46.270227 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:46.270229 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:46.270348 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:46.270708 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:46.270717 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:46.270719 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:46.271029 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:09:46.271343 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0926 15:09:46.271576 (+ 233us) server_negotiation.cc:299] Negotiation successful
0926 15:09:46.271800 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 15:09:46.272761 20501 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:09:46.393920 12427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8uoJGv/build/asan/bin/kudu with pid 20371
I20260926 15:09:46.415752 12427 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 19989
I20260926 15:09:46.416891 20061 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:09:46.529196 12427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8uoJGv/build/asan/bin/kudu with pid 19989
2026-09-26T15:09:46Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (7142 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790435318278681-12427-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 15:09:46 dist-test-slave-865v krb5kdc[20654](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 15:09:46 dist-test-slave-865v krb5kdc[20654](info): set up 2 sockets
Sep 26 15:09:46 dist-test-slave-865v krb5kdc[20654](info): commencing operation
krb5kdc: starting...
W20260926 15:09:47.641971 12427 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.011s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790435318278681-12427-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790435318278681-12427-0/krb5kdc/test-user.keytab.
Sep 26 15:09:47 dist-test-slave-865v krb5kdc[20654](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435387, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T15:09:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T15:09:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.34.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.34.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.34.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.254.keytab.
Entry for principal kudu/127.12.34.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.254.keytab.
I20260926 15:09:47.852633 12427 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8uoJGv/build/asan/bin/kudu
/tmp/dist-test-task8uoJGv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790435318278681-12427-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790435318278681-12427-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790435318278681-12427-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.34.254:46459
--webserver_interface=127.12.34.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.254.keytab
--principal=kudu/127.12.34.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.34.212:43713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.34.254:46459
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790435318278681-12427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790435318278681-12427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790435318278681-12427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:09:48.162863 20667 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:09:48.163115 20667 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:09:48.163275 20667 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:09:48.170269 20667 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:09:48.170368 20667 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:09:48.170469 20667 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:09:48.170560 20667 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:09:48.170645 20667 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:09:48.179895 20667 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.34.212:43713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790435318278681-12427-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790435318278681-12427-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.34.254:46459
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.254.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.34.254:46459
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.34.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790435318278681-12427-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:09:48.181384 20667 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:09:48.182832 20667 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:09:48.196069 20672 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:09:48.195996 20673 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:09:48.196228 20675 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:09:48.197000 20667 server_base.cc:1034] running on GCE node
I20260926 15:09:48.197821 20667 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:09:48.199173 20667 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:09:48.200452 20667 hybrid_clock.cc:648] HybridClock initialized: now 1790435388200416 us; error 44 us; skew 500 ppm
I20260926 15:09:48.200860 20667 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-26T15:09:48Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1738 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 15:09:48 dist-test-slave-865v krb5kdc[20681](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 15:09:48 dist-test-slave-865v krb5kdc[20681](info): set up 2 sockets
Sep 26 15:09:48 dist-test-slave-865v krb5kdc[20681](info): commencing operation
krb5kdc: starting...
W20260926 15:09:49.365803 12427 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.000s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/krb5kdc/test-user.keytab.
Sep 26 15:09:49 dist-test-slave-865v krb5kdc[20681](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435389, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T15:09:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T15:09:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.34.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.34.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.34.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.254.keytab.
Entry for principal kudu/127.12.34.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.254.keytab.
I20260926 15:09:49.570393 12427 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8uoJGv/build/asan/bin/kudu
/tmp/dist-test-task8uoJGv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.34.254:33193
--webserver_interface=127.12.34.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.254.keytab
--principal=kudu/127.12.34.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.34.212:39759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.34.254:33193
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:09:49.877898 20694 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:09:49.878233 20694 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:09:49.878417 20694 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:09:49.885414 20694 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:09:49.885563 20694 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:09:49.885670 20694 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:09:49.885774 20694 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:09:49.885866 20694 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:09:49.895260 20694 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.34.212:39759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.34.254:33193
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.254.keytab
--principal=kudu/127.12.34.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.34.254:33193
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.34.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:09:49.896533 20694 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:09:49.897783 20694 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:09:49.909662 20699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:09:49.909765 20700 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:09:49.909725 20702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:09:49.910581 20694 server_base.cc:1034] running on GCE node
I20260926 15:09:49.911298 20694 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:09:49.912588 20694 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:09:49.913781 20694 hybrid_clock.cc:648] HybridClock initialized: now 1790435389913735 us; error 38 us; skew 500 ppm
I20260926 15:09:49.914134 20694 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:09:49 dist-test-slave-865v krb5kdc[20681](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435389, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:09:49.924208 20694 init.cc:377] Logged in from keytab as kudu/127.12.34.254@KRBTEST.COM (short username kudu)
I20260926 15:09:49.925856 20694 webserver.cc:460] Webserver started at http://127.12.34.254:35091/ using document root <none> and password file <none>
I20260926 15:09:49.926539 20694 fs_manager.cc:359] Metadata directory not provided
I20260926 15:09:49.926682 20694 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:09:49.927014 20694 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:09:49.930058 20694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/minicluster-data/master-0/data/instance:
uuid: "5fa4d3a7179b48fa8311543a297aab44"
format_stamp: "Formatted at 2026-09-26 15:09:49 on dist-test-slave-865v"
server_key: "311ab8d102c54b4d3f0efe1ece8d7a29"
server_key_iv: "c8c7371c3b2d6737074587e78116ec79"
server_key_version: "encryptionkey@0"
I20260926 15:09:49.930791 20694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/minicluster-data/master-0/wal/instance:
uuid: "5fa4d3a7179b48fa8311543a297aab44"
format_stamp: "Formatted at 2026-09-26 15:09:49 on dist-test-slave-865v"
server_key: "311ab8d102c54b4d3f0efe1ece8d7a29"
server_key_iv: "c8c7371c3b2d6737074587e78116ec79"
server_key_version: "encryptionkey@0"
I20260926 15:09:49.935837 20694 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 15:09:49.939276 20709 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:09:49.940557 20694 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 15:09:49.940733 20694 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/minicluster-data/master-0/data,/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/minicluster-data/master-0/wal
uuid: "5fa4d3a7179b48fa8311543a297aab44"
format_stamp: "Formatted at 2026-09-26 15:09:49 on dist-test-slave-865v"
server_key: "311ab8d102c54b4d3f0efe1ece8d7a29"
server_key_iv: "c8c7371c3b2d6737074587e78116ec79"
server_key_version: "encryptionkey@0"
I20260926 15:09:49.940933 20694 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:09:49.956198 20694 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:09:49.960022 20694 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:09:49.960309 20694 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:09:49.978566 20694 rpc_server.cc:307] RPC server started. Bound to: 127.12.34.254:33193
I20260926 15:09:49.978605 20760 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.34.254:33193 every 8 connection(s)
I20260926 15:09:49.979879 20694 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
I20260926 15:09:49.983408 20761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:09:49.989490 12427 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8uoJGv/build/asan/bin/kudu as pid 20694
I20260926 15:09:49.989698 12427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/minicluster-data/master-0/wal/instance
I20260926 15:09:49.990542 12427 external_mini_cluster.cc:1428] Setting key 1b3092fb28ef61671524d434e4a75003
I20260926 15:09:49.994060 20761 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5fa4d3a7179b48fa8311543a297aab44: Bootstrap starting.
I20260926 15:09:49.997355 20761 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5fa4d3a7179b48fa8311543a297aab44: Neither blocks nor log segments found. Creating new log.
I20260926 15:09:49.998383 20761 log.cc:824] T 00000000000000000000000000000000 P 5fa4d3a7179b48fa8311543a297aab44: Log is configured to *not* fsync() on all Append() calls
I20260926 15:09:50.001294 20761 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5fa4d3a7179b48fa8311543a297aab44: No bootstrap required, opened a new log
Sep 26 15:09:49 dist-test-slave-865v krb5kdc[20681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435389, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.34.254@KRBTEST.COM
I20260926 15:09:50.011587 20761 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5fa4d3a7179b48fa8311543a297aab44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fa4d3a7179b48fa8311543a297aab44" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 33193 } }
I20260926 15:09:50.012053 20761 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5fa4d3a7179b48fa8311543a297aab44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:09:50.012144 20761 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5fa4d3a7179b48fa8311543a297aab44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fa4d3a7179b48fa8311543a297aab44, State: Initialized, Role: FOLLOWER
I20260926 15:09:50.012768 20761 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5fa4d3a7179b48fa8311543a297aab44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fa4d3a7179b48fa8311543a297aab44" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 33193 } }
I20260926 15:09:50.013238 20761 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5fa4d3a7179b48fa8311543a297aab44 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 15:09:50.013398 20761 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5fa4d3a7179b48fa8311543a297aab44 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 15:09:50.013571 20761 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5fa4d3a7179b48fa8311543a297aab44 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:09:50.014915 20764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:49.992548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46832 (local address 127.12.34.254:33193)
0926 15:09:49.993324 (+ 776us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:49.993335 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:49.993373 (+ 38us) server_negotiation.cc:407] Connection header received
0926 15:09:49.995084 (+ 1711us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:49.995104 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:49.995462 (+ 358us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:49.995764 (+ 302us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:09:49.997569 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:49.999026 (+ 1457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:50.000638 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:50.001350 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:50.001535 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:50.005986 (+ 4451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:09:50.005996 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:09:50.006015 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:09:50.006062 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:09:50.010291 (+ 4229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:50.011007 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:50.011015 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:50.011023 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:50.011228 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:50.011703 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:50.011708 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:50.011711 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:50.012389 (+ 678us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:09:50.012661 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0926 15:09:50.013127 (+ 466us) server_negotiation.cc:299] Negotiation successful
0926 15:09:50.013550 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":583,"thread_start_us":372,"threads_started":1}
I20260926 15:09:50.016289 20761 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5fa4d3a7179b48fa8311543a297aab44 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fa4d3a7179b48fa8311543a297aab44" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 33193 } }
I20260926 15:09:50.016758 20761 leader_election.cc:302] T 00000000000000000000000000000000 P 5fa4d3a7179b48fa8311543a297aab44 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5fa4d3a7179b48fa8311543a297aab44; no voters:
I20260926 15:09:50.017409 20761 leader_election.cc:288] T 00000000000000000000000000000000 P 5fa4d3a7179b48fa8311543a297aab44 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 15:09:50.017520 20766 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5fa4d3a7179b48fa8311543a297aab44 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:09:50.018603 20766 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5fa4d3a7179b48fa8311543a297aab44 [term 1 LEADER]: Becoming Leader. State: Replica: 5fa4d3a7179b48fa8311543a297aab44, State: Running, Role: LEADER
I20260926 15:09:50.019115 20766 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5fa4d3a7179b48fa8311543a297aab44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fa4d3a7179b48fa8311543a297aab44" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 33193 } }
I20260926 15:09:50.019618 20761 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5fa4d3a7179b48fa8311543a297aab44 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:09:50.022251 20768 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5fa4d3a7179b48fa8311543a297aab44 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5fa4d3a7179b48fa8311543a297aab44. Latest consensus state: current_term: 1 leader_uuid: "5fa4d3a7179b48fa8311543a297aab44" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fa4d3a7179b48fa8311543a297aab44" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 33193 } } }
I20260926 15:09:50.022367 20767 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5fa4d3a7179b48fa8311543a297aab44 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5fa4d3a7179b48fa8311543a297aab44" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fa4d3a7179b48fa8311543a297aab44" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 33193 } } }
I20260926 15:09:50.022760 20768 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5fa4d3a7179b48fa8311543a297aab44 [sys.catalog]: This master's current role is: LEADER
I20260926 15:09:50.022782 20767 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5fa4d3a7179b48fa8311543a297aab44 [sys.catalog]: This master's current role is: LEADER
I20260926 15:09:50.024547 20772 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:09:50.029695 20772 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 15:09:50.038695 20772 catalog_manager.cc:1347] Generated new cluster ID: 34baf286568d40939e0137a8fe8d3410
I20260926 15:09:50.038806 20772 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 15:09:50.067137 20772 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 15:09:50.068499 20772 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:09:50.079998 20772 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5fa4d3a7179b48fa8311543a297aab44: Generated new TSK 0
I20260926 15:09:50.080925 20772 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.34.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.34.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.34.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.193.keytab.
Entry for principal kudu/127.12.34.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.193.keytab.
I20260926 15:09:50.151851 12427 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8uoJGv/build/asan/bin/kudu
/tmp/dist-test-task8uoJGv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.34.193:0
--local_ip_for_outbound_sockets=127.12.34.193
--webserver_interface=127.12.34.193
--webserver_port=0
--tserver_master_addrs=127.12.34.254:33193
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.193.keytab
--principal=kudu/127.12.34.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.34.212:39759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:09:50.506616 20787 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:09:50.506924 20787 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:09:50.507088 20787 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:09:50.514467 20787 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:09:50.514629 20787 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:09:50.514840 20787 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.34.193
I20260926 15:09:50.524618 20787 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.34.212:39759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.193.keytab
--principal=kudu/127.12.34.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.34.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.34.193
--webserver_port=0
--tserver_master_addrs=127.12.34.254:33193
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.34.193
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:09:50.525938 20787 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:09:50.527202 20787 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:09:50.539752 20792 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:09:50.539726 20793 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:09:50.539685 20795 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:09:50.540540 20787 server_base.cc:1034] running on GCE node
I20260926 15:09:50.541163 20787 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:09:50.542050 20787 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:09:50.543278 20787 hybrid_clock.cc:648] HybridClock initialized: now 1790435390543237 us; error 57 us; skew 500 ppm
I20260926 15:09:50.543584 20787 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:09:50 dist-test-slave-865v krb5kdc[20681](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435390, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:09:50.546856 20787 init.cc:377] Logged in from keytab as kudu/127.12.34.193@KRBTEST.COM (short username kudu)
I20260926 15:09:50.548187 20787 webserver.cc:460] Webserver started at http://127.12.34.193:40181/ using document root <none> and password file <none>
I20260926 15:09:50.548843 20787 fs_manager.cc:359] Metadata directory not provided
I20260926 15:09:50.548985 20787 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:09:50.549316 20787 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:09:50.552019 20787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/minicluster-data/ts-0/data/instance:
uuid: "f5948441923d4ee3a00491be6d2fa5e6"
format_stamp: "Formatted at 2026-09-26 15:09:50 on dist-test-slave-865v"
server_key: "1715df548c977741307605c99ef4787c"
server_key_iv: "7bdb4ea002da4d0141ef9d5cf440ff5b"
server_key_version: "encryptionkey@0"
I20260926 15:09:50.552779 20787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/minicluster-data/ts-0/wal/instance:
uuid: "f5948441923d4ee3a00491be6d2fa5e6"
format_stamp: "Formatted at 2026-09-26 15:09:50 on dist-test-slave-865v"
server_key: "1715df548c977741307605c99ef4787c"
server_key_iv: "7bdb4ea002da4d0141ef9d5cf440ff5b"
server_key_version: "encryptionkey@0"
I20260926 15:09:50.557677 20787 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 15:09:50.560909 20802 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:09:50.562242 20787 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 15:09:50.562431 20787 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/minicluster-data/ts-0/data,/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/minicluster-data/ts-0/wal
uuid: "f5948441923d4ee3a00491be6d2fa5e6"
format_stamp: "Formatted at 2026-09-26 15:09:50 on dist-test-slave-865v"
server_key: "1715df548c977741307605c99ef4787c"
server_key_iv: "7bdb4ea002da4d0141ef9d5cf440ff5b"
server_key_version: "encryptionkey@0"
I20260926 15:09:50.562625 20787 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:09:50.581018 20787 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:09:50.585260 20787 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:09:50.585537 20787 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:09:50.586459 20787 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:09:50.587713 20787 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:09:50.587813 20787 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:09:50.587955 20787 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:09:50.588024 20787 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:09:50.621614 20787 rpc_server.cc:307] RPC server started. Bound to: 127.12.34.193:39325
I20260926 15:09:50.621662 20914 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.34.193:39325 every 8 connection(s)
I20260926 15:09:50.622829 20787 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/minicluster-data/ts-0/data/info.pb
I20260926 15:09:50.626632 12427 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8uoJGv/build/asan/bin/kudu as pid 20787
I20260926 15:09:50.626825 12427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/minicluster-data/ts-0/wal/instance
I20260926 15:09:50.627499 12427 external_mini_cluster.cc:1428] Setting key 3d3ff57ea6bd5d6b1a5c2fe3b4de5256
Sep 26 15:09:50 dist-test-slave-865v krb5kdc[20681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435390, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.193@KRBTEST.COM for kudu/127.12.34.254@KRBTEST.COM
I20260926 15:09:50.642794 20917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:50.625640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.193:47003 (local address 127.12.34.254:33193)
0926 15:09:50.626071 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:50.626075 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:50.627040 (+ 965us) server_negotiation.cc:407] Connection header received
0926 15:09:50.628797 (+ 1757us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:50.628802 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:50.628878 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:50.628993 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:09:50.630432 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:50.631648 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:50.633164 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:50.633863 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:50.634008 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:50.637566 (+ 3558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:09:50.637571 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:09:50.637574 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:09:50.637603 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:09:50.639934 (+ 2331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:50.640616 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:50.640629 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:50.640632 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:50.640734 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:50.641169 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:50.641174 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:50.641177 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:50.641439 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:09:50.641589 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0926 15:09:50.642227 (+ 638us) server_negotiation.cc:299] Negotiation successful
0926 15:09:50.642442 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":232,"threads_started":1}
I20260926 15:09:50.643682 20918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:50.626101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:33193 (local address 127.12.34.193:47003)
0926 15:09:50.626881 (+ 780us) negotiation.cc:107] Waiting for socket to connect
0926 15:09:50.626925 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 15:09:50.628563 (+ 1638us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:09:50.629248 (+ 685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:09:50.629256 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:09:50.629729 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:09:50.630256 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:50.630272 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:50.631812 (+ 1540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:50.631816 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:50.632999 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:50.633010 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:50.634059 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:50.634064 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:50.634242 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:50.634917 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:09:50.634937 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:09:50.637373 (+ 2436us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:09:50.640069 (+ 2696us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:09:50.640077 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:09:50.640092 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:09:50.640493 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:09:50.640839 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:09:50.640852 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:09:50.640856 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:09:50.641022 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:09:50.641563 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:09:50.641571 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:09:50.641875 (+ 304us) client_negotiation.cc:769] Sending connection context
0926 15:09:50.642123 (+ 248us) client_negotiation.cc:240] Negotiation successful
0926 15:09:50.642590 (+ 467us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":331,"threads_started":1}
I20260926 15:09:50.645717 20915 heartbeater.cc:347] Connected to a master server at 127.12.34.254:33193
I20260926 15:09:50.646028 20915 heartbeater.cc:464] Registering TS with master...
I20260926 15:09:50.646724 20915 heartbeater.cc:511] Master 127.12.34.254:33193 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.12.34.194@KRBTEST.COM; defaulting to no policy
I20260926 15:09:50.649107 20726 ts_manager.cc:194] Registered new tserver with Master: f5948441923d4ee3a00491be6d2fa5e6 (127.12.34.193:39325)
I20260926 15:09:50.650637 20726 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.34.193@KRBTEST.COM'} at 127.12.34.193:47003
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.34.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.34.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.194.keytab.
Entry for principal kudu/127.12.34.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.194.keytab.
I20260926 15:09:50.677418 12427 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8uoJGv/build/asan/bin/kudu
/tmp/dist-test-task8uoJGv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.34.194:0
--local_ip_for_outbound_sockets=127.12.34.194
--webserver_interface=127.12.34.194
--webserver_port=0
--tserver_master_addrs=127.12.34.254:33193
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.194.keytab
--principal=kudu/127.12.34.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.34.212:39759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:09:51.015271 20921 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:09:51.015518 20921 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:09:51.015691 20921 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:09:51.022593 20921 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:09:51.022680 20921 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:09:51.022873 20921 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.34.194
I20260926 15:09:51.032136 20921 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.34.212:39759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.194.keytab
--principal=kudu/127.12.34.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.34.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.34.194
--webserver_port=0
--tserver_master_addrs=127.12.34.254:33193
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.34.194
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:09:51.033367 20921 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:09:51.034600 20921 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:09:51.047176 20926 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:09:51.047214 20927 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:09:51.047117 20929 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:09:51.048038 20921 server_base.cc:1034] running on GCE node
I20260926 15:09:51.048604 20921 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:09:51.049556 20921 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:09:51.050746 20921 hybrid_clock.cc:648] HybridClock initialized: now 1790435391050711 us; error 33 us; skew 500 ppm
I20260926 15:09:51.051047 20921 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:09:51 dist-test-slave-865v krb5kdc[20681](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435391, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:09:51.054154 20921 init.cc:377] Logged in from keytab as kudu/127.12.34.194@KRBTEST.COM (short username kudu)
I20260926 15:09:51.055606 20921 webserver.cc:460] Webserver started at http://127.12.34.194:40589/ using document root <none> and password file <none>
I20260926 15:09:51.056263 20921 fs_manager.cc:359] Metadata directory not provided
I20260926 15:09:51.056360 20921 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:09:51.056684 20921 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:09:51.059453 20921 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/minicluster-data/ts-1/data/instance:
uuid: "d9de7496ee7b4f88b514447b14dc455c"
format_stamp: "Formatted at 2026-09-26 15:09:51 on dist-test-slave-865v"
server_key: "64f37acbb442943071ea49b9e821efd7"
server_key_iv: "aeb2085364111ee3dbf1beac72b8e58a"
server_key_version: "encryptionkey@0"
I20260926 15:09:51.060246 20921 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/minicluster-data/ts-1/wal/instance:
uuid: "d9de7496ee7b4f88b514447b14dc455c"
format_stamp: "Formatted at 2026-09-26 15:09:51 on dist-test-slave-865v"
server_key: "64f37acbb442943071ea49b9e821efd7"
server_key_iv: "aeb2085364111ee3dbf1beac72b8e58a"
server_key_version: "encryptionkey@0"
I20260926 15:09:51.065039 20921 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260926 15:09:51.068642 20936 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:09:51.070048 20921 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 15:09:51.070176 20921 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/minicluster-data/ts-1/data,/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/minicluster-data/ts-1/wal
uuid: "d9de7496ee7b4f88b514447b14dc455c"
format_stamp: "Formatted at 2026-09-26 15:09:51 on dist-test-slave-865v"
server_key: "64f37acbb442943071ea49b9e821efd7"
server_key_iv: "aeb2085364111ee3dbf1beac72b8e58a"
server_key_version: "encryptionkey@0"
I20260926 15:09:51.070335 20921 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:09:51.094141 20921 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:09:51.098336 20921 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:09:51.098573 20921 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:09:51.099587 20921 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:09:51.101259 20921 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:09:51.101336 20921 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:09:51.101455 20921 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:09:51.101509 20921 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:09:51.132818 20921 rpc_server.cc:307] RPC server started. Bound to: 127.12.34.194:43743
I20260926 15:09:51.132885 21048 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.34.194:43743 every 8 connection(s)
I20260926 15:09:51.134048 20921 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/minicluster-data/ts-1/data/info.pb
I20260926 15:09:51.138151 12427 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8uoJGv/build/asan/bin/kudu as pid 20921
I20260926 15:09:51.138339 12427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/minicluster-data/ts-1/wal/instance
I20260926 15:09:51.138799 12427 external_mini_cluster.cc:1428] Setting key 4ed950e19e68be1a5bc06393c20bc5fd
Sep 26 15:09:51 dist-test-slave-865v krb5kdc[20681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435391, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.194@KRBTEST.COM for kudu/127.12.34.254@KRBTEST.COM
I20260926 15:09:51.153506 20917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:51.136537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.194:49749 (local address 127.12.34.254:33193)
0926 15:09:51.136742 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:51.136748 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:51.137941 (+ 1193us) server_negotiation.cc:407] Connection header received
0926 15:09:51.139996 (+ 2055us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:51.140000 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:51.140062 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:51.140148 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:09:51.141817 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:51.142495 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:51.144023 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:51.144431 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:51.144563 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:51.147974 (+ 3411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:09:51.147979 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:09:51.147983 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:09:51.148011 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:09:51.150614 (+ 2603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:51.151330 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:51.151337 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:51.151346 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:51.151459 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:51.151871 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:51.151876 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:51.151879 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:51.152127 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:09:51.152285 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0926 15:09:51.152916 (+ 631us) server_negotiation.cc:299] Negotiation successful
0926 15:09:51.153177 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 15:09:51.154371 21051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:51.137050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:33193 (local address 127.12.34.194:49749)
0926 15:09:51.137763 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0926 15:09:51.137820 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0926 15:09:51.139696 (+ 1876us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:09:51.140402 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:09:51.140411 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:09:51.141061 (+ 650us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:09:51.141654 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:51.141669 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:51.142641 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:51.142647 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:51.143880 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:51.143892 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:51.144643 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:51.144650 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:51.144913 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:51.145678 (+ 765us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:09:51.145699 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:09:51.147758 (+ 2059us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:09:51.150766 (+ 3008us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:09:51.150775 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:09:51.150789 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:09:51.151182 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:09:51.151566 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:09:51.151572 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:09:51.151576 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:09:51.151756 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:09:51.152250 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:09:51.152258 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:09:51.152569 (+ 311us) client_negotiation.cc:769] Sending connection context
0926 15:09:51.152822 (+ 253us) client_negotiation.cc:240] Negotiation successful
0926 15:09:51.153327 (+ 505us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":249,"threads_started":1}
I20260926 15:09:51.155922 21049 heartbeater.cc:347] Connected to a master server at 127.12.34.254:33193
I20260926 15:09:51.156301 21049 heartbeater.cc:464] Registering TS with master...
I20260926 15:09:51.157009 21049 heartbeater.cc:511] Master 127.12.34.254:33193 requested a full tablet report, sending...
I20260926 15:09:51.158641 20726 ts_manager.cc:194] Registered new tserver with Master: d9de7496ee7b4f88b514447b14dc455c (127.12.34.194:43743)
WARNING: no policy specified for kudu/127.12.34.195@KRBTEST.COM; defaulting to no policy
I20260926 15:09:51.159798 20726 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.34.194@KRBTEST.COM'} at 127.12.34.194:49749
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.34.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.34.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.195.keytab.
Entry for principal kudu/127.12.34.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.195.keytab.
I20260926 15:09:51.188319 12427 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8uoJGv/build/asan/bin/kudu
/tmp/dist-test-task8uoJGv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.34.195:0
--local_ip_for_outbound_sockets=127.12.34.195
--webserver_interface=127.12.34.195
--webserver_port=0
--tserver_master_addrs=127.12.34.254:33193
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.195.keytab
--principal=kudu/127.12.34.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.34.212:39759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:09:51.498003 21054 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:09:51.498306 21054 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:09:51.498469 21054 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:09:51.505471 21054 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:09:51.505703 21054 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:09:51.505945 21054 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.34.195
I20260926 15:09:51.515808 21054 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.34.212:39759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.195.keytab
--principal=kudu/127.12.34.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.34.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.34.195
--webserver_port=0
--tserver_master_addrs=127.12.34.254:33193
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.34.195
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:09:51.517062 21054 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:09:51.518350 21054 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:09:51.531344 21059 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:09:51.531430 21060 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:09:51.531391 21062 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:09:51.534466 21054 server_base.cc:1034] running on GCE node
I20260926 15:09:51.535199 21054 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:09:51.536170 21054 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:09:51.537377 21054 hybrid_clock.cc:648] HybridClock initialized: now 1790435391537309 us; error 64 us; skew 500 ppm
I20260926 15:09:51.537659 21054 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:09:51 dist-test-slave-865v krb5kdc[20681](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435391, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:09:51.540755 21054 init.cc:377] Logged in from keytab as kudu/127.12.34.195@KRBTEST.COM (short username kudu)
I20260926 15:09:51.542243 21054 webserver.cc:460] Webserver started at http://127.12.34.195:45921/ using document root <none> and password file <none>
I20260926 15:09:51.542935 21054 fs_manager.cc:359] Metadata directory not provided
I20260926 15:09:51.543061 21054 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:09:51.543377 21054 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:09:51.546238 21054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/minicluster-data/ts-2/data/instance:
uuid: "6a9571f6698a4b9da14a98d6a60771d3"
format_stamp: "Formatted at 2026-09-26 15:09:51 on dist-test-slave-865v"
server_key: "be00001e6777ee10736eed5f34d31347"
server_key_iv: "c98934c4c5953c71c9d5a2de21963fe1"
server_key_version: "encryptionkey@0"
I20260926 15:09:51.546983 21054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/minicluster-data/ts-2/wal/instance:
uuid: "6a9571f6698a4b9da14a98d6a60771d3"
format_stamp: "Formatted at 2026-09-26 15:09:51 on dist-test-slave-865v"
server_key: "be00001e6777ee10736eed5f34d31347"
server_key_iv: "c98934c4c5953c71c9d5a2de21963fe1"
server_key_version: "encryptionkey@0"
I20260926 15:09:51.551762 21054 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.005s
I20260926 15:09:51.555071 21069 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:09:51.556363 21054 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 15:09:51.556541 21054 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/minicluster-data/ts-2/data,/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/minicluster-data/ts-2/wal
uuid: "6a9571f6698a4b9da14a98d6a60771d3"
format_stamp: "Formatted at 2026-09-26 15:09:51 on dist-test-slave-865v"
server_key: "be00001e6777ee10736eed5f34d31347"
server_key_iv: "c98934c4c5953c71c9d5a2de21963fe1"
server_key_version: "encryptionkey@0"
I20260926 15:09:51.556754 21054 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:09:51.580335 21054 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:09:51.584977 21054 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:09:51.585321 21054 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:09:51.586283 21054 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:09:51.587535 21054 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:09:51.587605 21054 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:09:51.587739 21054 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:09:51.587798 21054 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:09:51.620919 21054 rpc_server.cc:307] RPC server started. Bound to: 127.12.34.195:41957
I20260926 15:09:51.621057 21181 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.34.195:41957 every 8 connection(s)
I20260926 15:09:51.622225 21054 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/minicluster-data/ts-2/data/info.pb
I20260926 15:09:51.626626 12427 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8uoJGv/build/asan/bin/kudu as pid 21054
I20260926 15:09:51.626803 12427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/minicluster-data/ts-2/wal/instance
I20260926 15:09:51.627276 12427 external_mini_cluster.cc:1428] Setting key 942a2a344d5dc43a5944c7751ef9396d
Sep 26 15:09:51 dist-test-slave-865v krb5kdc[20681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435391, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.195@KRBTEST.COM for kudu/127.12.34.254@KRBTEST.COM
I20260926 15:09:51.641170 20917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:51.624946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.195:45539 (local address 127.12.34.254:33193)
0926 15:09:51.625161 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:51.625167 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:51.626358 (+ 1191us) server_negotiation.cc:407] Connection header received
0926 15:09:51.628499 (+ 2141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:51.628504 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:51.628602 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:51.628715 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:09:51.630277 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:51.631014 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:51.632417 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:51.632798 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:51.632901 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:51.636039 (+ 3138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:09:51.636044 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:09:51.636047 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:09:51.636076 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:09:51.638517 (+ 2441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:51.639144 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:51.639148 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:51.639150 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:51.639280 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:51.639643 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:51.639646 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:51.639648 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:51.639886 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:09:51.639991 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 15:09:51.640593 (+ 602us) server_negotiation.cc:299] Negotiation successful
0926 15:09:51.640819 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 15:09:51.642216 21184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:51.625464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:33193 (local address 127.12.34.195:45539)
0926 15:09:51.626212 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0926 15:09:51.626254 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 15:09:51.628232 (+ 1978us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:09:51.628933 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:09:51.628946 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:09:51.629509 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:09:51.630139 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:51.630158 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:51.631126 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:51.631129 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:51.632283 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:51.632293 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:51.632983 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:51.632988 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:51.633224 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:51.633832 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:09:51.633852 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:09:51.635861 (+ 2009us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:09:51.638639 (+ 2778us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:09:51.638645 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:09:51.638655 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:09:51.639036 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:09:51.639372 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:09:51.639375 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:09:51.639377 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:09:51.639535 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:09:51.639990 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:09:51.639995 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:09:51.640233 (+ 238us) client_negotiation.cc:769] Sending connection context
0926 15:09:51.640480 (+ 247us) client_negotiation.cc:240] Negotiation successful
0926 15:09:51.640888 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":289,"threads_started":1}
I20260926 15:09:51.643349 21182 heartbeater.cc:347] Connected to a master server at 127.12.34.254:33193
I20260926 15:09:51.643620 21182 heartbeater.cc:464] Registering TS with master...
I20260926 15:09:51.644313 21182 heartbeater.cc:511] Master 127.12.34.254:33193 requested a full tablet report, sending...
I20260926 15:09:51.645579 20726 ts_manager.cc:194] Registered new tserver with Master: 6a9571f6698a4b9da14a98d6a60771d3 (127.12.34.195:41957)
I20260926 15:09:51.646292 20726 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.34.195@KRBTEST.COM'} at 127.12.34.195:45539
I20260926 15:09:51.653939 20915 heartbeater.cc:503] Master 127.12.34.254:33193 was elected leader, sending a full tablet report...
I20260926 15:09:51.654325 12427 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.12.34.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.34.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.34.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/krb5kdc/oryx_127.12.34.193.keytab.
Entry for principal oryx/127.12.34.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/krb5kdc/oryx_127.12.34.193.keytab.
WARNING: no policy specified for oryx/127.12.34.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.34.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.34.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/krb5kdc/oryx_127.12.34.194.keytab.
Entry for principal oryx/127.12.34.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/krb5kdc/oryx_127.12.34.194.keytab.
WARNING: no policy specified for oryx/127.12.34.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.34.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.34.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/krb5kdc/oryx_127.12.34.195.keytab.
Entry for principal oryx/127.12.34.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/krb5kdc/oryx_127.12.34.195.keytab.
I20260926 15:09:51.782004 12427 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 20787
Sep 26 15:09:51 dist-test-slave-865v krb5kdc[20681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435389, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.34.193@KRBTEST.COM
I20260926 15:09:51.797246 21192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:51.783131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34802 (local address 127.12.34.193:39325)
0926 15:09:51.783767 (+ 636us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:51.783774 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:51.783800 (+ 26us) server_negotiation.cc:407] Connection header received
0926 15:09:51.783862 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:51.783868 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:51.784079 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:51.784210 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:09:51.785489 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:51.786485 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:51.787673 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:51.788198 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:51.788318 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:51.792196 (+ 3878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:09:51.792206 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:09:51.792219 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:09:51.792273 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:09:51.794835 (+ 2562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:51.795374 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:51.795380 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:51.795382 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:51.795493 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:51.795824 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:51.795828 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:51.795830 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:51.796169 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:09:51.796412 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0926 15:09:51.796675 (+ 263us) server_negotiation.cc:299] Negotiation successful
0926 15:09:51.796859 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":483,"thread_start_us":331,"threads_started":1}
I20260926 15:09:51.798502 20910 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:09:51.880492 12427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8uoJGv/build/asan/bin/kudu with pid 20787
I20260926 15:09:51.896603 12427 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 20921
Sep 26 15:09:51 dist-test-slave-865v krb5kdc[20681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435389, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.34.194@KRBTEST.COM
I20260926 15:09:51.912410 21194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:51.897475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60122 (local address 127.12.34.194:43743)
0926 15:09:51.897962 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:51.897969 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:51.897995 (+ 26us) server_negotiation.cc:407] Connection header received
0926 15:09:51.898061 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:51.898066 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:51.898272 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:51.898406 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:09:51.899546 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:51.900494 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:51.902140 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:51.902719 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:51.902833 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:51.906870 (+ 4037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:09:51.906885 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:09:51.906901 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:09:51.906933 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:09:51.910047 (+ 3114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:51.910673 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:51.910679 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:51.910681 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:51.910794 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:51.911094 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:51.911097 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:51.911099 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:51.911422 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:09:51.911638 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0926 15:09:51.911940 (+ 302us) server_negotiation.cc:299] Negotiation successful
0926 15:09:51.912114 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":257,"threads_started":1}
I20260926 15:09:51.913782 21044 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:09:51.995417 12427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8uoJGv/build/asan/bin/kudu with pid 20921
I20260926 15:09:52.011193 12427 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 21054
Sep 26 15:09:52 dist-test-slave-865v krb5kdc[20681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435389, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.34.195@KRBTEST.COM
I20260926 15:09:52.028379 21196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:52.012380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39982 (local address 127.12.34.195:41957)
0926 15:09:52.012979 (+ 599us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:52.012986 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:52.013015 (+ 29us) server_negotiation.cc:407] Connection header received
0926 15:09:52.013071 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:52.013165 (+ 94us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:52.013449 (+ 284us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:52.013641 (+ 192us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:09:52.015128 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:52.016127 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:52.017476 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:52.018009 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:52.018171 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:52.022571 (+ 4400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:09:52.022581 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:09:52.022594 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:09:52.022628 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:09:52.025570 (+ 2942us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:52.026226 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:52.026233 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:52.026236 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:52.026359 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:52.026718 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:52.026724 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:52.026727 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:52.027048 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:09:52.027379 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0926 15:09:52.027680 (+ 301us) server_negotiation.cc:299] Negotiation successful
0926 15:09:52.028042 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":467,"thread_start_us":309,"threads_started":1}
I20260926 15:09:52.029868 21177 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:09:52.112399 12427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8uoJGv/build/asan/bin/kudu with pid 21054
I20260926 15:09:52.128335 12427 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 20694
I20260926 15:09:52.129547 20756 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:09:52.232851 12427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8uoJGv/build/asan/bin/kudu with pid 20694
I20260926 15:09:52.252620 12427 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8uoJGv/build/asan/bin/kudu
/tmp/dist-test-task8uoJGv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.34.254:33193
--webserver_interface=127.12.34.254
--webserver_port=35091
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.254.keytab
--principal=kudu/127.12.34.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.34.212:39759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.34.254:33193
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:09:52.565055 21199 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:09:52.565335 21199 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:09:52.565485 21199 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:09:52.575346 21199 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:09:52.575438 21199 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:09:52.575536 21199 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:09:52.575630 21199 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:09:52.575704 21199 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:09:52.585239 21199 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.34.212:39759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.34.254:33193
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.254.keytab
--principal=kudu/127.12.34.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.34.254:33193
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.34.254
--webserver_port=35091
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:09:52.586378 21199 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:09:52.587579 21199 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:09:52.598723 21204 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:09:52.598659 21207 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:09:52.598834 21205 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:09:52.600101 21199 server_base.cc:1034] running on GCE node
I20260926 15:09:52.600859 21199 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:09:52.602202 21199 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:09:52.603472 21199 hybrid_clock.cc:648] HybridClock initialized: now 1790435392603392 us; error 63 us; skew 500 ppm
I20260926 15:09:52.603875 21199 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:09:52 dist-test-slave-865v krb5kdc[20681](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435392, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:09:52.607256 21199 init.cc:377] Logged in from keytab as kudu/127.12.34.254@KRBTEST.COM (short username kudu)
I20260926 15:09:52.608593 21199 webserver.cc:460] Webserver started at http://127.12.34.254:35091/ using document root <none> and password file <none>
I20260926 15:09:52.609251 21199 fs_manager.cc:359] Metadata directory not provided
I20260926 15:09:52.609359 21199 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:09:52.614318 21199 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 15:09:52.617141 21214 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:09:52.618554 21199 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 15:09:52.618716 21199 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/minicluster-data/master-0/data,/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/minicluster-data/master-0/wal
uuid: "5fa4d3a7179b48fa8311543a297aab44"
format_stamp: "Formatted at 2026-09-26 15:09:49 on dist-test-slave-865v"
server_key: "311ab8d102c54b4d3f0efe1ece8d7a29"
server_key_iv: "c8c7371c3b2d6737074587e78116ec79"
server_key_version: "encryptionkey@0"
I20260926 15:09:52.619640 21199 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:09:52.641433 21199 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:09:52.645622 21199 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:09:52.645884 21199 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:09:52.663192 21199 rpc_server.cc:307] RPC server started. Bound to: 127.12.34.254:33193
I20260926 15:09:52.663215 21265 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.34.254:33193 every 8 connection(s)
I20260926 15:09:52.664620 21199 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
I20260926 15:09:52.671255 21266 sys_catalog.cc:263] Verifying existing consensus state
I20260926 15:09:52.672067 12427 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8uoJGv/build/asan/bin/kudu as pid 21199
I20260926 15:09:52.672912 12427 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8uoJGv/build/asan/bin/kudu
/tmp/dist-test-task8uoJGv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.34.193:39325
--local_ip_for_outbound_sockets=127.12.34.193
--tserver_master_addrs=127.12.34.254:33193
--webserver_port=40181
--webserver_interface=127.12.34.193
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.193.keytab
--principal=kudu/127.12.34.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.34.212:39759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.12.34.193
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/krb5kdc/oryx_127.12.34.193.keytab with env {KRB5CCNAME=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 15:09:52.674497 21266 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5fa4d3a7179b48fa8311543a297aab44: Bootstrap starting.
I20260926 15:09:52.698942 21266 log.cc:824] T 00000000000000000000000000000000 P 5fa4d3a7179b48fa8311543a297aab44: Log is configured to *not* fsync() on all Append() calls
I20260926 15:09:52.709421 21266 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5fa4d3a7179b48fa8311543a297aab44: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 15:09:52.709995 21266 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5fa4d3a7179b48fa8311543a297aab44: Bootstrap complete.
I20260926 15:09:52.723033 21266 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5fa4d3a7179b48fa8311543a297aab44 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fa4d3a7179b48fa8311543a297aab44" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 33193 } }
I20260926 15:09:52.724162 21266 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5fa4d3a7179b48fa8311543a297aab44 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fa4d3a7179b48fa8311543a297aab44, State: Initialized, Role: FOLLOWER
I20260926 15:09:52.724818 21266 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5fa4d3a7179b48fa8311543a297aab44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fa4d3a7179b48fa8311543a297aab44" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 33193 } }
I20260926 15:09:52.725140 21266 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5fa4d3a7179b48fa8311543a297aab44 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 15:09:52.725270 21266 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5fa4d3a7179b48fa8311543a297aab44 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 15:09:52.725440 21266 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5fa4d3a7179b48fa8311543a297aab44 [term 1 FOLLOWER]: Advancing to term 2
I20260926 15:09:52.731478 21266 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5fa4d3a7179b48fa8311543a297aab44 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fa4d3a7179b48fa8311543a297aab44" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 33193 } }
I20260926 15:09:52.732059 21266 leader_election.cc:302] T 00000000000000000000000000000000 P 5fa4d3a7179b48fa8311543a297aab44 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5fa4d3a7179b48fa8311543a297aab44; no voters:
I20260926 15:09:52.732698 21266 leader_election.cc:288] T 00000000000000000000000000000000 P 5fa4d3a7179b48fa8311543a297aab44 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 15:09:52.732868 21270 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5fa4d3a7179b48fa8311543a297aab44 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 15:09:52.734097 21270 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5fa4d3a7179b48fa8311543a297aab44 [term 2 LEADER]: Becoming Leader. State: Replica: 5fa4d3a7179b48fa8311543a297aab44, State: Running, Role: LEADER
I20260926 15:09:52.734622 21270 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5fa4d3a7179b48fa8311543a297aab44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fa4d3a7179b48fa8311543a297aab44" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 33193 } }
I20260926 15:09:52.734799 21266 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5fa4d3a7179b48fa8311543a297aab44 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:09:52.737306 21271 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5fa4d3a7179b48fa8311543a297aab44 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "5fa4d3a7179b48fa8311543a297aab44" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fa4d3a7179b48fa8311543a297aab44" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 33193 } } }
I20260926 15:09:52.737735 21271 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5fa4d3a7179b48fa8311543a297aab44 [sys.catalog]: This master's current role is: LEADER
I20260926 15:09:52.742565 21272 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5fa4d3a7179b48fa8311543a297aab44 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5fa4d3a7179b48fa8311543a297aab44. Latest consensus state: current_term: 2 leader_uuid: "5fa4d3a7179b48fa8311543a297aab44" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fa4d3a7179b48fa8311543a297aab44" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 33193 } } }
I20260926 15:09:52.742950 21272 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5fa4d3a7179b48fa8311543a297aab44 [sys.catalog]: This master's current role is: LEADER
I20260926 15:09:52.746019 21276 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:09:52.751581 21276 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 15:09:52.752925 21276 catalog_manager.cc:1259] Loaded cluster ID: 34baf286568d40939e0137a8fe8d3410
I20260926 15:09:52.753033 21276 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 15:09:52.757063 21276 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:09:52.758467 21276 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 5fa4d3a7179b48fa8311543a297aab44: Loaded TSK: 0
I20260926 15:09:52.759907 21276 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 15:09:53.021972 21268 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:09:53.022267 21268 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:09:53.022459 21268 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:09:53.029664 21268 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:09:53.029798 21268 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:09:53.029963 21268 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.34.193
I20260926 15:09:53.039624 21268 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.34.212:39759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/krb5kdc/oryx_127.12.34.193.keytab
--principal=oryx/127.12.34.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.34.193:39325
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.34.193
--webserver_port=40181
--tserver_master_addrs=127.12.34.254:33193
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.34.193
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:09:53.040856 21268 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:09:53.042129 21268 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:09:53.054806 21292 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:09:53.054911 21293 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:09:53.054884 21295 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:09:53.056013 21268 server_base.cc:1034] running on GCE node
I20260926 15:09:53.056636 21268 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:09:53.057603 21268 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:09:53.058830 21268 hybrid_clock.cc:648] HybridClock initialized: now 1790435393058772 us; error 51 us; skew 500 ppm
I20260926 15:09:53.059131 21268 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:09:53 dist-test-slave-865v krb5kdc[20681](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435393, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.34.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:09:53.062463 21268 init.cc:377] Logged in from keytab as oryx/127.12.34.193@KRBTEST.COM (short username oryx)
I20260926 15:09:53.063970 21268 webserver.cc:460] Webserver started at http://127.12.34.193:40181/ using document root <none> and password file <none>
I20260926 15:09:53.064654 21268 fs_manager.cc:359] Metadata directory not provided
I20260926 15:09:53.064811 21268 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:09:53.070277 21268 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 15:09:53.073143 21302 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:09:53.074486 21268 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 15:09:53.074663 21268 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/minicluster-data/ts-0/data,/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/minicluster-data/ts-0/wal
uuid: "f5948441923d4ee3a00491be6d2fa5e6"
format_stamp: "Formatted at 2026-09-26 15:09:50 on dist-test-slave-865v"
server_key: "1715df548c977741307605c99ef4787c"
server_key_iv: "7bdb4ea002da4d0141ef9d5cf440ff5b"
server_key_version: "encryptionkey@0"
I20260926 15:09:53.075585 21268 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:09:53.096356 21268 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:09:53.100520 21268 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:09:53.100783 21268 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:09:53.101787 21268 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:09:53.103070 21268 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:09:53.103190 21268 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:09:53.103329 21268 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:09:53.103417 21268 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:09:53.134999 21268 rpc_server.cc:307] RPC server started. Bound to: 127.12.34.193:39325
I20260926 15:09:53.135080 21414 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.34.193:39325 every 8 connection(s)
I20260926 15:09:53.136417 21268 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/minicluster-data/ts-0/data/info.pb
I20260926 15:09:53.138096 12427 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8uoJGv/build/asan/bin/kudu as pid 21268
I20260926 15:09:53.138871 12427 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8uoJGv/build/asan/bin/kudu
/tmp/dist-test-task8uoJGv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.34.194:43743
--local_ip_for_outbound_sockets=127.12.34.194
--tserver_master_addrs=127.12.34.254:33193
--webserver_port=40589
--webserver_interface=127.12.34.194
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.194.keytab
--principal=kudu/127.12.34.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.34.212:39759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.12.34.194
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/krb5kdc/oryx_127.12.34.194.keytab with env {KRB5CCNAME=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 15:09:53 dist-test-slave-865v krb5kdc[20681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.34.193@KRBTEST.COM for oryx/127.12.34.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 15:09:53 dist-test-slave-865v krb5kdc[20681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.34.193@KRBTEST.COM for oryx/127.12.34.254@KRBTEST.COM, Server not found in Kerberos database
W20260926 15:09:53.462008 21419 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:09:53.462253 21419 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:09:53.462402 21419 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:09:53.469511 21419 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:09:53.469633 21419 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:09:53.469825 21419 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.34.194
I20260926 15:09:53.479292 21419 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.34.212:39759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/krb5kdc/oryx_127.12.34.194.keytab
--principal=oryx/127.12.34.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.34.194:43743
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.34.194
--webserver_port=40589
--tserver_master_addrs=127.12.34.254:33193
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.34.194
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:09:53.480566 21419 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:09:53.481848 21419 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:09:53.499706 21426 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:09:53.499739 21425 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:09:53.500901 21428 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:09:53.501348 21419 server_base.cc:1034] running on GCE node
I20260926 15:09:53.502347 21419 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:09:53.503876 21419 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:09:53.505159 21419 hybrid_clock.cc:648] HybridClock initialized: now 1790435393505081 us; error 49 us; skew 500 ppm
I20260926 15:09:53.505667 21419 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:09:53 dist-test-slave-865v krb5kdc[20681](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435393, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.34.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:09:53.511051 21419 init.cc:377] Logged in from keytab as oryx/127.12.34.194@KRBTEST.COM (short username oryx)
I20260926 15:09:53.513417 21419 webserver.cc:460] Webserver started at http://127.12.34.194:40589/ using document root <none> and password file <none>
I20260926 15:09:53.514601 21419 fs_manager.cc:359] Metadata directory not provided
I20260926 15:09:53.514813 21419 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:09:53.524423 21419 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.006s sys 0.000s
I20260926 15:09:53.528676 21435 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:09:53.530764 21419 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 15:09:53.530983 21419 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/minicluster-data/ts-1/data,/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/minicluster-data/ts-1/wal
uuid: "d9de7496ee7b4f88b514447b14dc455c"
format_stamp: "Formatted at 2026-09-26 15:09:51 on dist-test-slave-865v"
server_key: "64f37acbb442943071ea49b9e821efd7"
server_key_iv: "aeb2085364111ee3dbf1beac72b8e58a"
server_key_version: "encryptionkey@0"
I20260926 15:09:53.532856 21419 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:09:53.573239 21419 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:09:53.580931 21419 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:09:53.581389 21419 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:09:53.582964 21419 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:09:53.585446 21419 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:09:53.585618 21419 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:09:53.585757 21419 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:09:53.585876 21419 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:09:53.628556 21547 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.34.194:43743 every 8 connection(s)
I20260926 15:09:53.628548 21419 rpc_server.cc:307] RPC server started. Bound to: 127.12.34.194:43743
I20260926 15:09:53.631158 21419 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/minicluster-data/ts-1/data/info.pb
I20260926 15:09:53.632674 12427 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8uoJGv/build/asan/bin/kudu as pid 21419
I20260926 15:09:53.633407 12427 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8uoJGv/build/asan/bin/kudu
/tmp/dist-test-task8uoJGv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.34.195:41957
--local_ip_for_outbound_sockets=127.12.34.195
--tserver_master_addrs=127.12.34.254:33193
--webserver_port=45921
--webserver_interface=127.12.34.195
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.195.keytab
--principal=kudu/127.12.34.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.34.212:39759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.12.34.195
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/krb5kdc/oryx_127.12.34.195.keytab with env {KRB5CCNAME=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 15:09:53 dist-test-slave-865v krb5kdc[20681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.34.194@KRBTEST.COM for oryx/127.12.34.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 15:09:53 dist-test-slave-865v krb5kdc[20681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.34.194@KRBTEST.COM for oryx/127.12.34.254@KRBTEST.COM, Server not found in Kerberos database
W20260926 15:09:54.253808 21417 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.104s user 0.000s sys 0.013s
I20260926 15:09:54.255704 21417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:53.139742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:33193 (local address 127.12.34.193:33065)
0926 15:09:53.140541 (+ 799us) negotiation.cc:107] Waiting for socket to connect
0926 15:09:53.140589 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 15:09:53.142546 (+ 1957us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:09:53.143827 (+ 1281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:09:53.143840 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:09:53.144428 (+ 588us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:09:53.144919 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:53.144935 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:53.146501 (+ 1566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:53.146509 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:53.147834 (+ 1325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:53.147855 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:53.148820 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:53.148827 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:53.149138 (+ 311us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:53.149830 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:09:53.149856 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:09:54.254587 (+1104731us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.34.254:33193: Server oryx/127.12.34.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":324,"threads_started":1}
W20260926 15:09:54.255697 21418 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:09:53.139811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.193:33065 (local address 127.12.34.254:33193)
0926 15:09:53.141580 (+ 1769us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:53.141596 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:53.141668 (+ 72us) server_negotiation.cc:407] Connection header received
0926 15:09:53.143058 (+ 1390us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:53.143070 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:53.143366 (+ 296us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:53.143689 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:09:53.145134 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:53.146363 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:53.147976 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:53.148588 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:53.148781 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:54.254957 (+1106176us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.34.193:33065: BlockingRecv error: recv got EOF from 127.12.34.193:33065 (error 108)
Metrics: {"server-negotiator.queue_time_us":1500,"thread_start_us":1285,"threads_started":1}
W20260926 15:09:54.256883 21415 heartbeater.cc:644] Failed to heartbeat to 127.12.34.254:33193 (0 consecutive failures): Not authorized: Failed to ping master at 127.12.34.254:33193: Client connection negotiation failed: client connection to 127.12.34.254:33193: Server oryx/127.12.34.254@KRBTEST.COM not found in Kerberos database
Sep 26 15:09:54 dist-test-slave-865v krb5kdc[20681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.34.193@KRBTEST.COM for oryx/127.12.34.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 15:09:54 dist-test-slave-865v krb5kdc[20681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.34.193@KRBTEST.COM for oryx/127.12.34.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 15:09:54.266225 21417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:54.257617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:33193 (local address 127.12.34.193:58717)
0926 15:09:54.257887 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0926 15:09:54.257902 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 15:09:54.258044 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:09:54.258386 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:09:54.258391 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:09:54.258764 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:09:54.258919 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:54.258927 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:54.260004 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:54.260009 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:54.260934 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:54.260946 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:54.261670 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:54.261677 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:54.261849 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:54.262467 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:09:54.262483 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:09:54.265942 (+ 3459us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.34.254:33193: Server oryx/127.12.34.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":151}
W20260926 15:09:54.266366 21418 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:09:54.257645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.193:58717 (local address 127.12.34.254:33193)
0926 15:09:54.257807 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:54.257813 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:54.257975 (+ 162us) server_negotiation.cc:407] Connection header received
0926 15:09:54.258183 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:54.258187 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:54.258252 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:54.258359 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:09:54.259059 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:54.259863 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:54.261110 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:54.261495 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:54.261639 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:54.266117 (+ 4478us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.34.193:58717: BlockingRecv error: recv got EOF from 127.12.34.193:58717 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 26 15:09:54 dist-test-slave-865v krb5kdc[20681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.34.193@KRBTEST.COM for oryx/127.12.34.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 15:09:54 dist-test-slave-865v krb5kdc[20681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.34.193@KRBTEST.COM for oryx/127.12.34.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 15:09:54.275427 21417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:54.267186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:33193 (local address 127.12.34.193:50337)
0926 15:09:54.267391 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0926 15:09:54.267408 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 15:09:54.267507 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:09:54.267837 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:09:54.267841 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:09:54.268132 (+ 291us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:09:54.268272 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:54.268279 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:54.269246 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:54.269251 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:54.270315 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:54.270326 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:54.271006 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:54.271013 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:54.271179 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:54.271731 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:09:54.271748 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:09:54.275159 (+ 3411us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.34.254:33193: Server oryx/127.12.34.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":116}
W20260926 15:09:54.275563 21418 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:09:54.267248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.193:50337 (local address 127.12.34.254:33193)
0926 15:09:54.267393 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:54.267399 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:54.267456 (+ 57us) server_negotiation.cc:407] Connection header received
0926 15:09:54.267632 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:54.267637 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:54.267707 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:54.267809 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:09:54.268400 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:54.269068 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:54.270456 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:54.270855 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:54.270978 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:54.275352 (+ 4374us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.34.193:50337: BlockingRecv error: recv got EOF from 127.12.34.193:50337 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260926 15:09:54.275763 21415 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 15:09:54.290092 21552 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:09:54.290601 21552 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:09:54.290844 21552 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:09:54.309526 21552 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:09:54.309741 21552 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:09:54.310039 21552 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.34.195
I20260926 15:09:54.334813 21552 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.34.212:39759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/krb5kdc/oryx_127.12.34.195.keytab
--principal=oryx/127.12.34.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.34.195:41957
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.34.195
--webserver_port=45921
--tserver_master_addrs=127.12.34.254:33193
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.34.195
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:09:54.337252 21552 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:09:54.339622 21552 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:09:54.357961 21558 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:09:54.357903 21559 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:09:54.358013 21561 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:09:54.359742 21552 server_base.cc:1034] running on GCE node
I20260926 15:09:54.360703 21552 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:09:54.362294 21552 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:09:54.363595 21552 hybrid_clock.cc:648] HybridClock initialized: now 1790435394363505 us; error 63 us; skew 500 ppm
I20260926 15:09:54.364102 21552 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:09:54 dist-test-slave-865v krb5kdc[20681](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435394, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.34.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:09:54.369478 21552 init.cc:377] Logged in from keytab as oryx/127.12.34.195@KRBTEST.COM (short username oryx)
I20260926 15:09:54.371760 21552 webserver.cc:460] Webserver started at http://127.12.34.195:45921/ using document root <none> and password file <none>
I20260926 15:09:54.372985 21552 fs_manager.cc:359] Metadata directory not provided
I20260926 15:09:54.373206 21552 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:09:54.382640 21552 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.007s sys 0.001s
I20260926 15:09:54.387338 21568 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:09:54.389596 21552 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.004s
I20260926 15:09:54.389863 21552 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/minicluster-data/ts-2/data,/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/minicluster-data/ts-2/wal
uuid: "6a9571f6698a4b9da14a98d6a60771d3"
format_stamp: "Formatted at 2026-09-26 15:09:51 on dist-test-slave-865v"
server_key: "be00001e6777ee10736eed5f34d31347"
server_key_iv: "c98934c4c5953c71c9d5a2de21963fe1"
server_key_version: "encryptionkey@0"
I20260926 15:09:54.391812 21552 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:09:54.415766 21552 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:09:54.422914 21552 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:09:54.423303 21552 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:09:54.424826 21552 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:09:54.427124 21552 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:09:54.427294 21552 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:09:54.427449 21552 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:09:54.427560 21552 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:09:54.467739 21680 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.34.195:41957 every 8 connection(s)
I20260926 15:09:54.467746 21552 rpc_server.cc:307] RPC server started. Bound to: 127.12.34.195:41957
I20260926 15:09:54.470402 21552 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790435318278681-12427-0/minicluster-data/ts-2/data/info.pb
I20260926 15:09:54.472635 12427 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8uoJGv/build/asan/bin/kudu as pid 21552
Sep 26 15:09:54 dist-test-slave-865v krb5kdc[20681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.34.195@KRBTEST.COM for oryx/127.12.34.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 15:09:54 dist-test-slave-865v krb5kdc[20681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.34.195@KRBTEST.COM for oryx/127.12.34.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 15:09:54.504997 21418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:54.473796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46844 (local address 127.12.34.254:33193)
0926 15:09:54.473983 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:54.473989 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:54.487832 (+ 13843us) server_negotiation.cc:407] Connection header received
0926 15:09:54.488004 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:54.488014 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:54.488135 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:54.488314 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:09:54.490255 (+ 1941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:54.491913 (+ 1658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:54.493732 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:54.494118 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:54.494251 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:54.496697 (+ 2446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:09:54.496705 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:09:54.496718 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:09:54.496758 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:09:54.500712 (+ 3954us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:54.501837 (+ 1125us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:54.501846 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:54.501854 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:54.502059 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:09:54.502648 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:09:54.502653 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:09:54.502657 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:09:54.503242 (+ 585us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:09:54.503585 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0926 15:09:54.504129 (+ 544us) server_negotiation.cc:299] Negotiation successful
0926 15:09:54.504534 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":69}
W20260926 15:09:54.892573 21550 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.241s user 0.003s sys 0.014s
W20260926 15:09:54.893887 21551 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:09:53.636836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.194:44133 (local address 127.12.34.254:33193)
0926 15:09:53.640558 (+ 3722us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:53.640567 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:53.640589 (+ 22us) server_negotiation.cc:407] Connection header received
0926 15:09:53.640675 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:53.640683 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:53.640768 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:53.640886 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:09:53.643796 (+ 2910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:53.645021 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:53.648405 (+ 3384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:53.649231 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:53.649377 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:54.893575 (+1244198us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.34.194:44133: BlockingRecv error: recv got EOF from 127.12.34.194:44133 (error 108)
Metrics: {"server-negotiator.queue_time_us":3571,"thread_start_us":3453,"threads_started":1}
I20260926 15:09:54.894428 21550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:53.636010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:33193 (local address 127.12.34.194:44133)
0926 15:09:53.636991 (+ 981us) negotiation.cc:107] Waiting for socket to connect
0926 15:09:53.637045 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 15:09:53.639844 (+ 2799us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:09:53.641337 (+ 1493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:09:53.641354 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:09:53.642440 (+ 1086us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:09:53.643539 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:53.643573 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:53.645230 (+ 1657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:53.645240 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:53.648217 (+ 2977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:53.648239 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:53.649567 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:53.649578 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:53.650003 (+ 425us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:53.651039 (+ 1036us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:09:53.651091 (+ 52us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:09:54.893389 (+1242298us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.34.254:33193: Server oryx/127.12.34.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":642,"thread_start_us":866,"threads_started":1}
W20260926 15:09:54.895407 21548 heartbeater.cc:644] Failed to heartbeat to 127.12.34.254:33193 (0 consecutive failures): Not authorized: Failed to ping master at 127.12.34.254:33193: Client connection negotiation failed: client connection to 127.12.34.254:33193: Server oryx/127.12.34.254@KRBTEST.COM not found in Kerberos database
Sep 26 15:09:54 dist-test-slave-865v krb5kdc[20681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.34.194@KRBTEST.COM for oryx/127.12.34.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 15:09:54 dist-test-slave-865v krb5kdc[20681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.34.194@KRBTEST.COM for oryx/127.12.34.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 15:09:54.904942 21550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:54.896029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:33193 (local address 127.12.34.194:43429)
0926 15:09:54.896213 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0926 15:09:54.896233 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 15:09:54.896341 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:09:54.896670 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:09:54.896675 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:09:54.897030 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:09:54.897241 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:54.897250 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:54.898254 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:54.898259 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:54.899345 (+ 1086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:54.899356 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:54.900118 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:54.900124 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:54.900310 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:54.900925 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:09:54.900952 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:09:54.904651 (+ 3699us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.34.254:33193: Server oryx/127.12.34.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":57}
W20260926 15:09:54.905069 21551 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:09:54.896135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.194:43429 (local address 127.12.34.254:33193)
0926 15:09:54.896308 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:54.896313 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:54.896329 (+ 16us) server_negotiation.cc:407] Connection header received
0926 15:09:54.896477 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:54.896482 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:54.896550 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:54.896648 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:09:54.897368 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:54.898112 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:54.899483 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:54.899955 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:54.900094 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:54.904836 (+ 4742us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.34.194:43429: BlockingRecv error: recv got EOF from 127.12.34.194:43429 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 26 15:09:54 dist-test-slave-865v krb5kdc[20681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.34.194@KRBTEST.COM for oryx/127.12.34.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 15:09:54 dist-test-slave-865v krb5kdc[20681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.34.194@KRBTEST.COM for oryx/127.12.34.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 15:09:54.916191 21550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:54.905654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:33193 (local address 127.12.34.194:37411)
0926 15:09:54.905795 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0926 15:09:54.905820 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 15:09:54.905921 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:09:54.906654 (+ 733us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:09:54.906658 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:09:54.906951 (+ 293us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:09:54.907095 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:54.907103 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:54.909136 (+ 2033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:54.909143 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:54.910056 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:54.910065 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:54.911313 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:54.911317 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:54.911426 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:54.912120 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:09:54.912136 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:09:54.916015 (+ 3879us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.34.254:33193: Server oryx/127.12.34.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":50}
W20260926 15:09:54.916594 21548 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 15:09:54.916769 21551 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:09:54.905998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.194:37411 (local address 127.12.34.254:33193)
0926 15:09:54.906227 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:54.906237 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:54.906262 (+ 25us) server_negotiation.cc:407] Connection header received
0926 15:09:54.906330 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:54.906339 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:54.906466 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:54.906613 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:09:54.907241 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:54.908903 (+ 1662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:54.910196 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:54.911157 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:54.911328 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:54.916370 (+ 5042us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.34.194:37411: BlockingRecv error: recv got EOF from 127.12.34.194:37411 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 26 15:09:55 dist-test-slave-865v krb5kdc[20681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.34.193@KRBTEST.COM for oryx/127.12.34.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 15:09:55 dist-test-slave-865v krb5kdc[20681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.34.193@KRBTEST.COM for oryx/127.12.34.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 15:09:55.291764 21687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:55.276426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:33193 (local address 127.12.34.193:52175)
0926 15:09:55.277119 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0926 15:09:55.277136 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 15:09:55.277261 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:09:55.277699 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:09:55.277702 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:09:55.278100 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:09:55.278244 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:55.278252 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:55.280222 (+ 1970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:55.280231 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:55.281273 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:55.281286 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:55.282603 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:55.282612 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:55.282800 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:55.287992 (+ 5192us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:09:55.288011 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:09:55.291537 (+ 3526us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.34.254:33193: Server oryx/127.12.34.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":424,"threads_started":1}
W20260926 15:09:55.292276 21551 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:09:55.276572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.193:52175 (local address 127.12.34.254:33193)
0926 15:09:55.276837 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:55.276847 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:55.277231 (+ 384us) server_negotiation.cc:407] Connection header received
0926 15:09:55.277387 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:55.277396 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:55.277526 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:55.277682 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:09:55.278388 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:55.280031 (+ 1643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:55.281432 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:55.282379 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:55.282555 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:55.291925 (+ 9370us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.34.193:52175: BlockingRecv error: recv got EOF from 127.12.34.193:52175 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
W20260926 15:09:55.590569 21685 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.091s user 0.000s sys 0.014s
W20260926 15:09:55.591960 21684 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:09:54.474236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.195:49433 (local address 127.12.34.254:33193)
0926 15:09:54.490130 (+ 15894us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:54.490138 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:54.490156 (+ 18us) server_negotiation.cc:407] Connection header received
0926 15:09:54.490210 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:54.490217 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:54.490286 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:54.490422 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:09:54.491949 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:54.493221 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:54.497166 (+ 3945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:54.497747 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:54.497894 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:55.591667 (+1093773us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.34.195:49433: BlockingRecv error: recv got EOF from 127.12.34.195:49433 (error 108)
Metrics: {"server-negotiator.queue_time_us":15775,"thread_start_us":15654,"threads_started":1}
I20260926 15:09:55.592417 21685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:54.474699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:33193 (local address 127.12.34.195:49433)
0926 15:09:54.486787 (+ 12088us) negotiation.cc:107] Waiting for socket to connect
0926 15:09:54.486834 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 15:09:54.489416 (+ 2582us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:09:54.490677 (+ 1261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:09:54.490688 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:09:54.491183 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:09:54.491695 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:54.491742 (+ 47us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:54.493417 (+ 1675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:54.493426 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:54.496390 (+ 2964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:54.496413 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:54.498634 (+ 2221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:54.498641 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:54.498996 (+ 355us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:54.499827 (+ 831us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:09:54.499859 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:09:55.591375 (+1091516us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.34.254:33193: Server oryx/127.12.34.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":11780,"thread_start_us":11553,"threads_started":1}
W20260926 15:09:55.593322 21681 heartbeater.cc:644] Failed to heartbeat to 127.12.34.254:33193 (0 consecutive failures): Not authorized: Failed to ping master at 127.12.34.254:33193: Client connection negotiation failed: client connection to 127.12.34.254:33193: Server oryx/127.12.34.254@KRBTEST.COM not found in Kerberos database
Sep 26 15:09:55 dist-test-slave-865v krb5kdc[20681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.34.195@KRBTEST.COM for oryx/127.12.34.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 15:09:55 dist-test-slave-865v krb5kdc[20681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.34.195@KRBTEST.COM for oryx/127.12.34.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 15:09:55.603089 21685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:55.594020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:33193 (local address 127.12.34.195:55095)
0926 15:09:55.594300 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0926 15:09:55.594321 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 15:09:55.594451 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:09:55.594781 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:09:55.594786 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:09:55.595155 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:09:55.595311 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:55.595318 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:55.596337 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:55.596343 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:55.597313 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:55.597324 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:55.598009 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:55.598014 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:55.598176 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:55.598814 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:09:55.598830 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:09:55.602784 (+ 3954us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.34.254:33193: Server oryx/127.12.34.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":89}
W20260926 15:09:55.603199 21684 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:09:55.594067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.195:55095 (local address 127.12.34.254:33193)
0926 15:09:55.594358 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:55.594363 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:55.594382 (+ 19us) server_negotiation.cc:407] Connection header received
0926 15:09:55.594596 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:55.594599 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:55.594659 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:55.594757 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:09:55.595458 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:55.596204 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:55.597448 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:55.597832 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:55.597979 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:55.602970 (+ 4991us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.34.195:55095: BlockingRecv error: recv got EOF from 127.12.34.195:55095 (error 108)
Metrics: {"server-negotiator.queue_time_us":177}
Sep 26 15:09:55 dist-test-slave-865v krb5kdc[20681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.34.195@KRBTEST.COM for oryx/127.12.34.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 15:09:55 dist-test-slave-865v krb5kdc[20681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.34.195@KRBTEST.COM for oryx/127.12.34.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 15:09:55.611366 21685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:55.603790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:33193 (local address 127.12.34.195:36591)
0926 15:09:55.604065 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0926 15:09:55.604079 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 15:09:55.604163 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:09:55.604414 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:09:55.604417 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:09:55.604679 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:09:55.604821 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:55.604828 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:55.605823 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:55.605826 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:55.606741 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:55.606751 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:55.607410 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:55.607413 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:55.607541 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:55.608098 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:09:55.608113 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:09:55.611186 (+ 3073us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.34.254:33193: Server oryx/127.12.34.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":50}
W20260926 15:09:55.611608 21684 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:09:55.603842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.195:36591 (local address 127.12.34.254:33193)
0926 15:09:55.603985 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:55.603990 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:55.604171 (+ 181us) server_negotiation.cc:407] Connection header received
0926 15:09:55.604235 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:55.604238 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:55.604293 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:55.604366 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:09:55.604938 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:55.605701 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:55.606856 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:55.607281 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:55.607373 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:55.611360 (+ 3987us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.34.195:36591: BlockingRecv error: recv got EOF from 127.12.34.195:36591 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260926 15:09:55.611703 21681 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 26 15:09:55 dist-test-slave-865v krb5kdc[20681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.34.194@KRBTEST.COM for oryx/127.12.34.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 15:09:55 dist-test-slave-865v krb5kdc[20681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.34.194@KRBTEST.COM for oryx/127.12.34.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 15:09:55.926504 21688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:55.917215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:33193 (local address 127.12.34.194:49171)
0926 15:09:55.917766 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0926 15:09:55.917793 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0926 15:09:55.917912 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:09:55.918215 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:09:55.918218 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:09:55.918623 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:09:55.918762 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:55.918769 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:55.919795 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:55.919799 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:55.921140 (+ 1341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:55.921159 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:55.922013 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:55.922021 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:55.922180 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:55.922890 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:09:55.922907 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:09:55.926250 (+ 3343us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.34.254:33193: Server oryx/127.12.34.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":322,"threads_started":1}
W20260926 15:09:55.926644 21684 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:09:55.917334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.194:49171 (local address 127.12.34.254:33193)
0926 15:09:55.917511 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:55.917515 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:55.917879 (+ 364us) server_negotiation.cc:407] Connection header received
0926 15:09:55.918008 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:55.918012 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:55.918085 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:55.918176 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:09:55.918896 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:55.919617 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:55.921287 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:55.921812 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:55.921942 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:55.926433 (+ 4491us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.34.194:49171: BlockingRecv error: recv got EOF from 127.12.34.194:49171 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 26 15:09:56 dist-test-slave-865v krb5kdc[20681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.34.193@KRBTEST.COM for oryx/127.12.34.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 15:09:56 dist-test-slave-865v krb5kdc[20681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.34.193@KRBTEST.COM for oryx/127.12.34.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 15:09:56.301682 21689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:56.292666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:33193 (local address 127.12.34.193:53903)
0926 15:09:56.293215 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0926 15:09:56.293232 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 15:09:56.293360 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:09:56.293643 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:09:56.293646 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:09:56.294041 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:09:56.294216 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:56.294224 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:56.295232 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:56.295236 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:56.296136 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:56.296147 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:56.296857 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:56.296862 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:56.297039 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:56.297753 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:09:56.297771 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:09:56.301454 (+ 3683us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.34.254:33193: Server oryx/127.12.34.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":285,"threads_started":1}
W20260926 15:09:56.301868 21684 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:09:56.292842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.193:53903 (local address 127.12.34.254:33193)
0926 15:09:56.293069 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:56.293103 (+ 34us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:56.293359 (+ 256us) server_negotiation.cc:407] Connection header received
0926 15:09:56.293462 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:56.293465 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:56.293526 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:56.293598 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:09:56.294364 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:56.295100 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:56.296254 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:56.296692 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:56.296787 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:56.301646 (+ 4859us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.34.193:53903: BlockingRecv error: recv got EOF from 127.12.34.193:53903 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Sep 26 15:09:56 dist-test-slave-865v krb5kdc[20681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.34.195@KRBTEST.COM for oryx/127.12.34.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 15:09:56 dist-test-slave-865v krb5kdc[20681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.34.195@KRBTEST.COM for oryx/127.12.34.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 15:09:56.621414 21690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:56.612276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:33193 (local address 127.12.34.195:60505)
0926 15:09:56.612850 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0926 15:09:56.612866 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 15:09:56.612981 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:09:56.613278 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:09:56.613281 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:09:56.613743 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:09:56.613883 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:56.613891 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:56.614983 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:56.614989 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:56.616227 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:56.616240 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:56.616971 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:56.616977 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:56.617157 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:56.617777 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:09:56.617794 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:09:56.621063 (+ 3269us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.34.254:33193: Server oryx/127.12.34.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":338,"threads_started":1}
W20260926 15:09:56.621539 21684 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:09:56.612324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.195:60505 (local address 127.12.34.254:33193)
0926 15:09:56.612495 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:56.612500 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:56.612940 (+ 440us) server_negotiation.cc:407] Connection header received
0926 15:09:56.613092 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:56.613095 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:56.613153 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:56.613230 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:09:56.614041 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:56.614818 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:56.616364 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:56.616789 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:56.616924 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:56.621310 (+ 4386us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.34.195:60505: BlockingRecv error: recv got EOF from 127.12.34.195:60505 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 26 15:09:56 dist-test-slave-865v krb5kdc[20681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.34.194@KRBTEST.COM for oryx/127.12.34.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 15:09:56 dist-test-slave-865v krb5kdc[20681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.34.194@KRBTEST.COM for oryx/127.12.34.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 15:09:56.936419 21691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:56.927379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:33193 (local address 127.12.34.194:50883)
0926 15:09:56.927892 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0926 15:09:56.927908 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 15:09:56.928029 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:09:56.928340 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:09:56.928343 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:09:56.928708 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:09:56.928903 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:56.928912 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:56.929940 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:56.929946 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:56.930854 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:56.930864 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:56.931674 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:56.931679 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:56.931805 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:56.932483 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:09:56.932502 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:09:56.936060 (+ 3558us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.34.254:33193: Server oryx/127.12.34.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":305,"threads_started":1}
W20260926 15:09:56.936641 21684 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:09:56.927520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.194:50883 (local address 127.12.34.254:33193)
0926 15:09:56.927687 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:56.927692 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:56.928007 (+ 315us) server_negotiation.cc:407] Connection header received
0926 15:09:56.928147 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:56.928151 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:56.928212 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:56.928294 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:09:56.929031 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:56.929750 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:56.931017 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:56.931502 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:56.931611 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:56.936338 (+ 4727us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.34.194:50883: BlockingRecv error: recv got EOF from 127.12.34.194:50883 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 26 15:09:57 dist-test-slave-865v krb5kdc[20681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.34.193@KRBTEST.COM for oryx/127.12.34.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 15:09:57 dist-test-slave-865v krb5kdc[20681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.34.193@KRBTEST.COM for oryx/127.12.34.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 15:09:57.311373 21692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:57.302542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:33193 (local address 127.12.34.193:60351)
0926 15:09:57.303077 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0926 15:09:57.303093 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 15:09:57.303202 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:09:57.303487 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:09:57.303490 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:09:57.303837 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:09:57.303994 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:57.304002 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:57.304973 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:57.304976 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:57.305913 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:57.305924 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:57.306764 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:57.306769 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:57.306971 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:57.307680 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:09:57.307710 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:09:57.311121 (+ 3411us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.34.254:33193: Server oryx/127.12.34.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":289,"threads_started":1}
W20260926 15:09:57.311563 21684 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:09:57.302756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.193:60351 (local address 127.12.34.254:33193)
0926 15:09:57.302937 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:57.302941 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:57.303173 (+ 232us) server_negotiation.cc:407] Connection header received
0926 15:09:57.303298 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:57.303301 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:57.303361 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:57.303444 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:09:57.304141 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:57.304846 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:57.306036 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:57.306618 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:57.306744 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:57.311251 (+ 4507us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.34.193:60351: BlockingRecv error: recv got EOF from 127.12.34.193:60351 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 26 15:09:57 dist-test-slave-865v krb5kdc[20681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.34.195@KRBTEST.COM for oryx/127.12.34.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 15:09:57 dist-test-slave-865v krb5kdc[20681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.34.195@KRBTEST.COM for oryx/127.12.34.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 15:09:57.631758 21693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:57.622412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:33193 (local address 127.12.34.195:40633)
0926 15:09:57.622928 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0926 15:09:57.622943 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 15:09:57.623108 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:09:57.623484 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:09:57.623489 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:09:57.623922 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:09:57.624093 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:57.624101 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:57.625286 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:57.625291 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:57.626646 (+ 1355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:57.626660 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:57.627331 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:57.627335 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:57.627467 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:57.628004 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:09:57.628020 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:09:57.631528 (+ 3508us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.34.254:33193: Server oryx/127.12.34.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":283,"threads_started":1}
W20260926 15:09:57.631918 21684 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:09:57.622485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.195:40633 (local address 127.12.34.254:33193)
0926 15:09:57.622776 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:57.622784 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:57.623034 (+ 250us) server_negotiation.cc:407] Connection header received
0926 15:09:57.623276 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:57.623281 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:57.623352 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:57.623455 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:09:57.624238 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:57.625136 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:57.626784 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:57.627195 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:57.627308 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:57.631686 (+ 4378us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.34.195:40633: BlockingRecv error: recv got EOF from 127.12.34.195:40633 (error 108)
Metrics: {"server-negotiator.queue_time_us":102}
Sep 26 15:09:57 dist-test-slave-865v krb5kdc[20681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.34.194@KRBTEST.COM for oryx/127.12.34.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 15:09:57 dist-test-slave-865v krb5kdc[20681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.34.194@KRBTEST.COM for oryx/127.12.34.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 15:09:57.946058 21694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:57.937375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:33193 (local address 127.12.34.194:50311)
0926 15:09:57.937850 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0926 15:09:57.937868 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 15:09:57.937983 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:09:57.938268 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:09:57.938272 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:09:57.938596 (+ 324us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:09:57.938777 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:57.938786 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:57.939722 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:57.939726 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:57.940724 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:57.940735 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:57.941495 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:57.941499 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:57.941629 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:57.942206 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:09:57.942223 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:09:57.945817 (+ 3594us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.34.254:33193: Server oryx/127.12.34.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":245,"threads_started":1}
W20260926 15:09:57.946230 21684 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:09:57.937424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.194:50311 (local address 127.12.34.254:33193)
0926 15:09:57.937647 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:57.937653 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:57.937952 (+ 299us) server_negotiation.cc:407] Connection header received
0926 15:09:57.938080 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:57.938083 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:57.938143 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:57.938222 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:09:57.938898 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:57.939593 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:57.940869 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:57.941351 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:57.941446 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:57.946010 (+ 4564us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.34.194:50311: BlockingRecv error: recv got EOF from 127.12.34.194:50311 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 26 15:09:58 dist-test-slave-865v krb5kdc[20681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.34.193@KRBTEST.COM for oryx/127.12.34.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 15:09:58 dist-test-slave-865v krb5kdc[20681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.34.193@KRBTEST.COM for oryx/127.12.34.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 15:09:58.320924 21695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:58.312255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:33193 (local address 127.12.34.193:39423)
0926 15:09:58.312782 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0926 15:09:58.312798 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 15:09:58.312945 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:09:58.313402 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:09:58.313407 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:09:58.313925 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:09:58.314157 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:58.314170 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:58.315167 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:58.315171 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:58.316134 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:58.316145 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:58.316833 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:58.316837 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:58.316959 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:58.317545 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:09:58.317561 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:09:58.320682 (+ 3121us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.34.254:33193: Server oryx/127.12.34.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":304,"threads_started":1}
W20260926 15:09:58.321146 21684 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:09:58.312439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.193:39423 (local address 127.12.34.254:33193)
0926 15:09:58.312636 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:58.312643 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:58.312893 (+ 250us) server_negotiation.cc:407] Connection header received
0926 15:09:58.313092 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:58.313096 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:58.313194 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:58.313358 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:09:58.314333 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:58.315043 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:58.316277 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:58.316712 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:58.316812 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:58.320883 (+ 4071us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.34.193:39423: BlockingRecv error: recv got EOF from 127.12.34.193:39423 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 26 15:09:58 dist-test-slave-865v krb5kdc[20681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.34.195@KRBTEST.COM for oryx/127.12.34.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 15:09:58 dist-test-slave-865v krb5kdc[20681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.34.195@KRBTEST.COM for oryx/127.12.34.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 15:09:58.641430 21696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:58.632631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:33193 (local address 127.12.34.195:39201)
0926 15:09:58.633245 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0926 15:09:58.633265 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 15:09:58.633404 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:09:58.633824 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:09:58.633830 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:09:58.634359 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:09:58.634585 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:58.634603 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:58.635729 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:58.635733 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:58.636678 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:58.636688 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:58.637308 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:58.637312 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:58.637432 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:58.637967 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:09:58.637982 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:09:58.641184 (+ 3202us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.34.254:33193: Server oryx/127.12.34.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":269,"threads_started":1}
W20260926 15:09:58.641615 21684 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:09:58.632771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.195:39201 (local address 127.12.34.254:33193)
0926 15:09:58.632933 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:58.632937 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:58.633357 (+ 420us) server_negotiation.cc:407] Connection header received
0926 15:09:58.633535 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:58.633541 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:58.633665 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:58.633771 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:09:58.634735 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:58.635596 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:58.636788 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:58.637194 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:58.637281 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:58.641369 (+ 4088us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.34.195:39201: BlockingRecv error: recv got EOF from 127.12.34.195:39201 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 26 15:09:58 dist-test-slave-865v krb5kdc[20681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.34.194@KRBTEST.COM for oryx/127.12.34.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 15:09:58 dist-test-slave-865v krb5kdc[20681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.34.194@KRBTEST.COM for oryx/127.12.34.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 15:09:58.956074 21697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:58.946982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:33193 (local address 127.12.34.194:38685)
0926 15:09:58.947699 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0926 15:09:58.947721 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 15:09:58.947839 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:09:58.948153 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:09:58.948157 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:09:58.948519 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:09:58.948672 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:58.948681 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:58.949718 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:58.949722 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:58.950732 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:58.950745 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:58.951427 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:58.951433 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:58.951564 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:58.952155 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:09:58.952172 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:09:58.955726 (+ 3554us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.34.254:33193: Server oryx/127.12.34.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":312,"threads_started":1}
W20260926 15:09:58.956137 21684 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:09:58.947054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.194:38685 (local address 127.12.34.254:33193)
0926 15:09:58.947269 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:58.947276 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:58.947833 (+ 557us) server_negotiation.cc:407] Connection header received
0926 15:09:58.947941 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:58.947944 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:58.948006 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:58.948088 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:09:58.948803 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:58.949586 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:58.950866 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:58.951263 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:58.951354 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:58.955906 (+ 4552us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.34.194:38685: BlockingRecv error: recv got EOF from 127.12.34.194:38685 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
Sep 26 15:09:59 dist-test-slave-865v krb5kdc[20681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.34.193@KRBTEST.COM for oryx/127.12.34.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 15:09:59 dist-test-slave-865v krb5kdc[20681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.34.193@KRBTEST.COM for oryx/127.12.34.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 15:09:59.330622 21698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:59.322045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:33193 (local address 127.12.34.193:47555)
0926 15:09:59.322596 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0926 15:09:59.322612 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 15:09:59.322761 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:09:59.323087 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:09:59.323092 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:09:59.323516 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:09:59.323656 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:59.323664 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:59.324677 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:59.324684 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:59.325614 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:59.325625 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:59.326272 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:59.326278 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:59.326467 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:59.327038 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:09:59.327054 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:09:59.330385 (+ 3331us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.34.254:33193: Server oryx/127.12.34.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":283,"threads_started":1}
W20260926 15:09:59.330866 21684 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:09:59.322089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.193:47555 (local address 127.12.34.254:33193)
0926 15:09:59.322295 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:59.322304 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:59.322699 (+ 395us) server_negotiation.cc:407] Connection header received
0926 15:09:59.322889 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:59.322894 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:59.322962 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:59.323060 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:09:59.323803 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:59.324542 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:59.325754 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:59.326137 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:59.326262 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:59.330595 (+ 4333us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.34.193:47555: BlockingRecv error: recv got EOF from 127.12.34.193:47555 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 26 15:09:59 dist-test-slave-865v krb5kdc[20681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.34.195@KRBTEST.COM for oryx/127.12.34.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 15:09:59 dist-test-slave-865v krb5kdc[20681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.34.195@KRBTEST.COM for oryx/127.12.34.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 15:09:59.651481 21699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:59.642522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:33193 (local address 127.12.34.195:46551)
0926 15:09:59.643168 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0926 15:09:59.643184 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 15:09:59.643331 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:09:59.643695 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:09:59.643700 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:09:59.644083 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:09:59.644236 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:59.644244 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:59.645256 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:59.645264 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:59.646220 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:59.646230 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:59.646943 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:59.646948 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:59.647133 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:59.647683 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:09:59.647700 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:09:59.651100 (+ 3400us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.34.254:33193: Server oryx/127.12.34.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":377,"threads_started":1}
W20260926 15:09:59.651573 21684 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:09:59.642653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.195:46551 (local address 127.12.34.254:33193)
0926 15:09:59.642967 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:59.642975 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:59.643270 (+ 295us) server_negotiation.cc:407] Connection header received
0926 15:09:59.643471 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:59.643475 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:59.643556 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:59.643657 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:09:59.644386 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:59.645130 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:59.646372 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:59.646806 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:59.646931 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:59.651282 (+ 4351us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.34.195:46551: BlockingRecv error: recv got EOF from 127.12.34.195:46551 (error 108)
Metrics: {"server-negotiator.queue_time_us":171}
Sep 26 15:09:59 dist-test-slave-865v krb5kdc[20681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.34.194@KRBTEST.COM for oryx/127.12.34.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 15:09:59 dist-test-slave-865v krb5kdc[20681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.34.194@KRBTEST.COM for oryx/127.12.34.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 15:09:59.965368 21700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:09:59.956996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:33193 (local address 127.12.34.194:37163)
0926 15:09:59.957516 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0926 15:09:59.957533 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 15:09:59.957645 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:09:59.957917 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:09:59.957920 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:09:59.958335 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:09:59.958474 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:59.958482 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:59.959598 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:59.959606 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:59.960527 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:09:59.960538 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:09:59.961258 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:09:59.961262 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:09:59.961393 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:59.961946 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:09:59.961963 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:09:59.965150 (+ 3187us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.34.254:33193: Server oryx/127.12.34.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":306,"threads_started":1}
W20260926 15:09:59.965513 21684 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:09:59.957149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.194:37163 (local address 127.12.34.254:33193)
0926 15:09:59.957360 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0926 15:09:59.957385 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0926 15:09:59.957611 (+ 226us) server_negotiation.cc:407] Connection header received
0926 15:09:59.957732 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:09:59.957736 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:09:59.957792 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:09:59.957871 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:09:59.958621 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:59.959468 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:59.960667 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:09:59.961066 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:09:59.961219 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:09:59.965304 (+ 4085us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.34.194:37163: BlockingRecv error: recv got EOF from 127.12.34.194:37163 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 26 15:10:00 dist-test-slave-865v krb5kdc[20681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.34.193@KRBTEST.COM for oryx/127.12.34.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 15:10:00 dist-test-slave-865v krb5kdc[20681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.34.193@KRBTEST.COM for oryx/127.12.34.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 15:10:00.345276 21701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:10:00.331618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:33193 (local address 127.12.34.193:47721)
0926 15:10:00.332142 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0926 15:10:00.332168 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0926 15:10:00.332299 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:10:00.332633 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:10:00.332638 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:10:00.333047 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:10:00.333335 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:10:00.333346 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:10:00.334587 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:10:00.334592 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:10:00.335523 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:10:00.335534 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:10:00.336394 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:10:00.336398 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:10:00.336520 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:10:00.341423 (+ 4903us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:10:00.341487 (+ 64us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:10:00.344984 (+ 3497us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.34.254:33193: Server oryx/127.12.34.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":274,"threads_started":1}
W20260926 15:10:00.345592 21684 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:10:00.331876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.193:47721 (local address 127.12.34.254:33193)
0926 15:10:00.332063 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0926 15:10:00.332069 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:10:00.332298 (+ 229us) server_negotiation.cc:407] Connection header received
0926 15:10:00.332384 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:10:00.332389 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:10:00.332452 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:10:00.332538 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:10:00.333485 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:10:00.334236 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:10:00.335709 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:10:00.336231 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:10:00.336355 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:10:00.345277 (+ 8922us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.34.193:47721: BlockingRecv error: recv got EOF from 127.12.34.193:47721 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 26 15:10:00 dist-test-slave-865v krb5kdc[20681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.34.195@KRBTEST.COM for oryx/127.12.34.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 15:10:00 dist-test-slave-865v krb5kdc[20681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.34.195@KRBTEST.COM for oryx/127.12.34.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 15:10:00.662205 21702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:10:00.652478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:33193 (local address 127.12.34.195:60679)
0926 15:10:00.652987 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0926 15:10:00.653003 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 15:10:00.653206 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:10:00.653619 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:10:00.653623 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:10:00.654114 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:10:00.654287 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:10:00.654297 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:10:00.655442 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:10:00.655446 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:10:00.656384 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:10:00.656395 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:10:00.657205 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:10:00.657210 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:10:00.657364 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:10:00.657927 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:10:00.657945 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:10:00.661923 (+ 3978us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.34.254:33193: Server oryx/127.12.34.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":269,"threads_started":1}
W20260926 15:10:00.662376 21684 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:10:00.652540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.195:60679 (local address 127.12.34.254:33193)
0926 15:10:00.652762 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0926 15:10:00.652768 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:10:00.653108 (+ 340us) server_negotiation.cc:407] Connection header received
0926 15:10:00.653355 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:10:00.653359 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:10:00.653428 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:10:00.653512 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:10:00.654489 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:10:00.655304 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:10:00.656557 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:10:00.657009 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:10:00.657132 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:10:00.662140 (+ 5008us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.34.195:60679: BlockingRecv error: recv got EOF from 127.12.34.195:60679 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 26 15:10:00 dist-test-slave-865v krb5kdc[20681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.34.194@KRBTEST.COM for oryx/127.12.34.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 15:10:00 dist-test-slave-865v krb5kdc[20681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.34.194@KRBTEST.COM for oryx/127.12.34.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 15:10:00.986702 21703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:10:00.966424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:33193 (local address 127.12.34.194:50567)
0926 15:10:00.966962 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0926 15:10:00.966992 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0926 15:10:00.967114 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:10:00.967455 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:10:00.967459 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:10:00.967855 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:10:00.968037 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:10:00.968046 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:10:00.969444 (+ 1398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:10:00.969451 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:10:00.970698 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:10:00.970711 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:10:00.971481 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:10:00.971487 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:10:00.971686 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:10:00.982333 (+ 10647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:10:00.982353 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:10:00.986377 (+ 4024us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.34.254:33193: Server oryx/127.12.34.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":304,"threads_started":1}
W20260926 15:10:00.986873 21684 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:10:00.966492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.194:50567 (local address 127.12.34.254:33193)
0926 15:10:00.966775 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0926 15:10:00.966782 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:10:00.967076 (+ 294us) server_negotiation.cc:407] Connection header received
0926 15:10:00.967238 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:10:00.967242 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:10:00.967309 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:10:00.967399 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:10:00.968190 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:10:00.969295 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:10:00.970865 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:10:00.971306 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:10:00.971464 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:10:00.986593 (+ 15129us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.34.194:50567: BlockingRecv error: recv got EOF from 127.12.34.194:50567 (error 108)
Metrics: {"server-negotiator.queue_time_us":104}
Sep 26 15:10:01 dist-test-slave-865v krb5kdc[20681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.34.193@KRBTEST.COM for oryx/127.12.34.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 15:10:01 dist-test-slave-865v krb5kdc[20681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.34.193@KRBTEST.COM for oryx/127.12.34.254@KRBTEST.COM, Server not found in Kerberos database
W20260926 15:10:01.358196 21684 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:10:01.346286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.193:36423 (local address 127.12.34.254:33193)
0926 15:10:01.346511 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0926 15:10:01.346520 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 15:10:01.346930 (+ 410us) server_negotiation.cc:407] Connection header received
0926 15:10:01.347107 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:10:01.347115 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:10:01.347216 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:10:01.347337 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:10:01.348123 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:10:01.349469 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:10:01.350810 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:10:01.351612 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:10:01.351909 (+ 297us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:10:01.358033 (+ 6124us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.34.193:36423: BlockingRecv error: recv got EOF from 127.12.34.193:36423 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 15:10:01.358345 21704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:10:01.346208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:33193 (local address 127.12.34.193:36423)
0926 15:10:01.346787 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0926 15:10:01.346803 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 15:10:01.346993 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:10:01.347370 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:10:01.347373 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:10:01.347747 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:10:01.347981 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:10:01.347992 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:10:01.349631 (+ 1639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:10:01.349635 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:10:01.350667 (+ 1032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:10:01.350678 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:10:01.351920 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:10:01.351926 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:10:01.352091 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:10:01.352960 (+ 869us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:10:01.352990 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:10:01.357880 (+ 4890us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.34.254:33193: Server oryx/127.12.34.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":277,"threads_started":1}
Sep 26 15:10:01 dist-test-slave-865v krb5kdc[20681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.34.195@KRBTEST.COM for oryx/127.12.34.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 15:10:01 dist-test-slave-865v krb5kdc[20681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.34.195@KRBTEST.COM for oryx/127.12.34.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 15:10:01.677110 21705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:10:01.663511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:33193 (local address 127.12.34.195:42481)
0926 15:10:01.664061 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0926 15:10:01.664084 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 15:10:01.664246 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:10:01.664585 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:10:01.664590 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:10:01.665222 (+ 632us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:10:01.665480 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:10:01.665493 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:10:01.667381 (+ 1888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:10:01.667390 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:10:01.669273 (+ 1883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:10:01.669291 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:10:01.670861 (+ 1570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:10:01.670869 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:10:01.671123 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:10:01.671941 (+ 818us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:10:01.671971 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:10:01.676769 (+ 4798us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.34.254:33193: Server oryx/127.12.34.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":300,"threads_started":1}
W20260926 15:10:01.677210 21684 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:10:01.663766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.195:42481 (local address 127.12.34.254:33193)
0926 15:10:01.663961 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0926 15:10:01.663969 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 15:10:01.664160 (+ 191us) server_negotiation.cc:407] Connection header received
0926 15:10:01.664385 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:10:01.664390 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:10:01.664473 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:10:01.664560 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:10:01.665642 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:10:01.667205 (+ 1563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:10:01.669466 (+ 2261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:10:01.670404 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:10:01.670524 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:10:01.676951 (+ 6427us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.34.195:42481: BlockingRecv error: recv got EOF from 127.12.34.195:42481 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 26 15:10:01 dist-test-slave-865v krb5kdc[20681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.34.194@KRBTEST.COM for oryx/127.12.34.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 15:10:01 dist-test-slave-865v krb5kdc[20681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.34.194@KRBTEST.COM for oryx/127.12.34.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 15:10:01.997560 21706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:10:01.987890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:33193 (local address 127.12.34.194:47767)
0926 15:10:01.988592 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0926 15:10:01.988608 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 15:10:01.988777 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:10:01.989178 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:10:01.989184 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:10:01.989732 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:10:01.989993 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:10:01.990006 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:10:01.991224 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:10:01.991228 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:10:01.992168 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:10:01.992179 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:10:01.993118 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:10:01.993128 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:10:01.993240 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:10:01.994032 (+ 792us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:10:01.994048 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:10:01.997333 (+ 3285us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.34.254:33193: Server oryx/127.12.34.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":607,"thread_start_us":354,"threads_started":1}
W20260926 15:10:01.997848 21684 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:10:01.988003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.194:47767 (local address 127.12.34.254:33193)
0926 15:10:01.988400 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0926 15:10:01.988406 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:10:01.988707 (+ 301us) server_negotiation.cc:407] Connection header received
0926 15:10:01.988917 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:10:01.988923 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:10:01.988999 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:10:01.989135 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:10:01.990192 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:10:01.991068 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:10:01.992326 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:10:01.992737 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:10:01.992854 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:10:01.997546 (+ 4692us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.34.194:47767: BlockingRecv error: recv got EOF from 127.12.34.194:47767 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
Sep 26 15:10:02 dist-test-slave-865v krb5kdc[20681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.34.193@KRBTEST.COM for oryx/127.12.34.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 15:10:02 dist-test-slave-865v krb5kdc[20681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.34.193@KRBTEST.COM for oryx/127.12.34.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 15:10:02.367990 21707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:10:02.359357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:33193 (local address 127.12.34.193:40473)
0926 15:10:02.359930 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0926 15:10:02.359946 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 15:10:02.360106 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:10:02.360468 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:10:02.360472 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:10:02.360883 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:10:02.361129 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:10:02.361143 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:10:02.362193 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:10:02.362197 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:10:02.363125 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:10:02.363136 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:10:02.363835 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:10:02.363840 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:10:02.363946 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:10:02.364522 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:10:02.364539 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:10:02.367743 (+ 3204us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.34.254:33193: Server oryx/127.12.34.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":281,"threads_started":1}
W20260926 15:10:02.368191 21684 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:10:02.359466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.193:40473 (local address 127.12.34.254:33193)
0926 15:10:02.359712 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0926 15:10:02.359718 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:10:02.360041 (+ 323us) server_negotiation.cc:407] Connection header received
0926 15:10:02.360249 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:10:02.360254 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:10:02.360341 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:10:02.360438 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:10:02.361285 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:10:02.362071 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:10:02.363254 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:10:02.363674 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:10:02.363766 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:10:02.367940 (+ 4174us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.34.193:40473: BlockingRecv error: recv got EOF from 127.12.34.193:40473 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Sep 26 15:10:02 dist-test-slave-865v krb5kdc[20681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.34.195@KRBTEST.COM for oryx/127.12.34.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 15:10:02 dist-test-slave-865v krb5kdc[20681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.34.195@KRBTEST.COM for oryx/127.12.34.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 15:10:02.687726 21708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:10:02.678321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:33193 (local address 127.12.34.195:53511)
0926 15:10:02.679056 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0926 15:10:02.679081 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 15:10:02.679219 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:10:02.679537 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:10:02.679542 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:10:02.679998 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:10:02.680248 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:10:02.680266 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:10:02.681406 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:10:02.681412 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:10:02.682626 (+ 1214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:10:02.682638 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:10:02.683342 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:10:02.683348 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:10:02.683506 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:10:02.684078 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:10:02.684095 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:10:02.687369 (+ 3274us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.34.254:33193: Server oryx/127.12.34.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":611,"thread_start_us":409,"threads_started":1}
W20260926 15:10:02.687748 21684 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:10:02.678383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.195:53511 (local address 127.12.34.254:33193)
0926 15:10:02.678593 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0926 15:10:02.678600 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:10:02.679151 (+ 551us) server_negotiation.cc:407] Connection header received
0926 15:10:02.679330 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:10:02.679335 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:10:02.679405 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:10:02.679508 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:10:02.680387 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:10:02.681269 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:10:02.682765 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:10:02.683177 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:10:02.683326 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:10:02.687530 (+ 4204us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.34.195:53511: BlockingRecv error: recv got EOF from 127.12.34.195:53511 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 26 15:10:03 dist-test-slave-865v krb5kdc[20681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.34.194@KRBTEST.COM for oryx/127.12.34.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 15:10:03 dist-test-slave-865v krb5kdc[20681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.34.194@KRBTEST.COM for oryx/127.12.34.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 15:10:03.008591 21709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:10:02.998503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:33193 (local address 127.12.34.194:36899)
0926 15:10:02.999077 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0926 15:10:02.999093 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 15:10:02.999402 (+ 309us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:10:02.999809 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:10:02.999814 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:10:03.000229 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:10:03.000391 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:10:03.000400 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:10:03.001622 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:10:03.001627 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:10:03.002579 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:10:03.002589 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:10:03.003327 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:10:03.003334 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:10:03.003493 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:10:03.004459 (+ 966us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:10:03.004476 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:10:03.008348 (+ 3872us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.34.254:33193: Server oryx/127.12.34.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":325,"threads_started":1}
W20260926 15:10:03.008810 21684 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:10:02.998602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.194:36899 (local address 127.12.34.254:33193)
0926 15:10:02.998821 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0926 15:10:02.998828 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:10:02.999235 (+ 407us) server_negotiation.cc:407] Connection header received
0926 15:10:02.999567 (+ 332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:10:02.999571 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:10:02.999665 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:10:02.999779 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:10:03.000531 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:10:03.001490 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:10:03.002732 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:10:03.003142 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:10:03.003269 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:10:03.008568 (+ 5299us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.34.194:36899: BlockingRecv error: recv got EOF from 127.12.34.194:36899 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 26 15:10:03 dist-test-slave-865v krb5kdc[20681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.34.193@KRBTEST.COM for oryx/127.12.34.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 15:10:03 dist-test-slave-865v krb5kdc[20681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.34.193@KRBTEST.COM for oryx/127.12.34.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 15:10:03.378589 21710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:10:03.369022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:33193 (local address 127.12.34.193:45135)
0926 15:10:03.369716 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0926 15:10:03.369733 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 15:10:03.369854 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:10:03.370179 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:10:03.370183 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:10:03.370560 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:10:03.370719 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:10:03.370728 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:10:03.371888 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:10:03.371892 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:10:03.372836 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:10:03.372848 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:10:03.373637 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:10:03.373642 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:10:03.373787 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:10:03.374486 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:10:03.374504 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:10:03.378270 (+ 3766us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.34.254:33193: Server oryx/127.12.34.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":569,"thread_start_us":420,"threads_started":1}
W20260926 15:10:03.378739 21684 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:10:03.369149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.193:45135 (local address 127.12.34.254:33193)
0926 15:10:03.369324 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0926 15:10:03.369328 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:10:03.369837 (+ 509us) server_negotiation.cc:407] Connection header received
0926 15:10:03.369970 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:10:03.369974 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:10:03.370036 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:10:03.370118 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:10:03.370873 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:10:03.371733 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:10:03.372980 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:10:03.373477 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:10:03.373597 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:10:03.378530 (+ 4933us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.34.193:45135: BlockingRecv error: recv got EOF from 127.12.34.193:45135 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 26 15:10:03 dist-test-slave-865v krb5kdc[20681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.34.195@KRBTEST.COM for oryx/127.12.34.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 15:10:03 dist-test-slave-865v krb5kdc[20681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.34.195@KRBTEST.COM for oryx/127.12.34.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 15:10:03.697570 21711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:10:03.688785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:33193 (local address 127.12.34.195:45413)
0926 15:10:03.689376 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0926 15:10:03.689397 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 15:10:03.689553 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:10:03.689912 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:10:03.689917 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:10:03.690362 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:10:03.690525 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:10:03.690533 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:10:03.691514 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:10:03.691521 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:10:03.692437 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:10:03.692448 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:10:03.693202 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:10:03.693208 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:10:03.693391 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:10:03.693973 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:10:03.693990 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:10:03.697327 (+ 3337us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.34.254:33193: Server oryx/127.12.34.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":317,"threads_started":1}
W20260926 15:10:03.697723 21684 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:10:03.688958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.195:45413 (local address 127.12.34.254:33193)
0926 15:10:03.689182 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0926 15:10:03.689188 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:10:03.689468 (+ 280us) server_negotiation.cc:407] Connection header received
0926 15:10:03.689692 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:10:03.689701 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:10:03.689785 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:10:03.689886 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:10:03.690665 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:10:03.691378 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:10:03.692572 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:10:03.693005 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:10:03.693162 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:10:03.697505 (+ 4343us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.34.195:45413: BlockingRecv error: recv got EOF from 127.12.34.195:45413 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Sep 26 15:10:04 dist-test-slave-865v krb5kdc[20681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.34.194@KRBTEST.COM for oryx/127.12.34.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 15:10:04 dist-test-slave-865v krb5kdc[20681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.34.194@KRBTEST.COM for oryx/127.12.34.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 15:10:04.018728 21712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:10:04.009754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:33193 (local address 127.12.34.194:34565)
0926 15:10:04.010313 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0926 15:10:04.010330 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 15:10:04.010438 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:10:04.010798 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:10:04.010804 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:10:04.011270 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:10:04.011444 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:10:04.011455 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:10:04.012452 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:10:04.012458 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:10:04.013469 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:10:04.013480 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:10:04.014153 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:10:04.014158 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:10:04.014333 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:10:04.014904 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:10:04.014921 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:10:04.018378 (+ 3457us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.34.254:33193: Server oryx/127.12.34.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":275,"threads_started":1}
W20260926 15:10:04.018815 21684 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:10:04.009795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.194:34565 (local address 127.12.34.254:33193)
0926 15:10:04.009991 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0926 15:10:04.009996 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:10:04.010394 (+ 398us) server_negotiation.cc:407] Connection header received
0926 15:10:04.010548 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:10:04.010554 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:10:04.010634 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:10:04.010750 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:10:04.011584 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:10:04.012315 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:10:04.013602 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:10:04.014005 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:10:04.014125 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:10:04.018569 (+ 4444us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.34.194:34565: BlockingRecv error: recv got EOF from 127.12.34.194:34565 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 26 15:10:04 dist-test-slave-865v krb5kdc[20681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.34.193@KRBTEST.COM for oryx/127.12.34.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 15:10:04 dist-test-slave-865v krb5kdc[20681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.34.193@KRBTEST.COM for oryx/127.12.34.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 15:10:04.388360 21713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:10:04.379618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:33193 (local address 127.12.34.193:40293)
0926 15:10:04.380072 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0926 15:10:04.380088 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 15:10:04.380200 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:10:04.380508 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:10:04.380512 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:10:04.380859 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:10:04.381009 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:10:04.381017 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:10:04.382176 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:10:04.382180 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:10:04.383157 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:10:04.383169 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:10:04.383893 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:10:04.383898 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:10:04.384012 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:10:04.384659 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:10:04.384676 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:10:04.388040 (+ 3364us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.34.254:33193: Server oryx/127.12.34.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":237,"threads_started":1}
W20260926 15:10:04.388530 21684 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:10:04.379722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.193:40293 (local address 127.12.34.254:33193)
0926 15:10:04.379961 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0926 15:10:04.379967 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:10:04.380181 (+ 214us) server_negotiation.cc:407] Connection header received
0926 15:10:04.380315 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:10:04.380319 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:10:04.380379 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:10:04.380458 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:10:04.381196 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:10:04.382051 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:10:04.383288 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:10:04.383707 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:10:04.383817 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:10:04.388245 (+ 4428us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.34.193:40293: BlockingRecv error: recv got EOF from 127.12.34.193:40293 (error 108)
Metrics: {"server-negotiator.queue_time_us":105}
Sep 26 15:10:04 dist-test-slave-865v krb5kdc[20681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.34.195@KRBTEST.COM for oryx/127.12.34.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 15:10:04 dist-test-slave-865v krb5kdc[20681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.34.195@KRBTEST.COM for oryx/127.12.34.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 15:10:04.707836 21714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:10:04.698668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:33193 (local address 127.12.34.195:34609)
0926 15:10:04.699240 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0926 15:10:04.699257 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 15:10:04.699365 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:10:04.699677 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:10:04.699685 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:10:04.700226 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:10:04.700392 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:10:04.700401 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:10:04.701805 (+ 1404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:10:04.701810 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:10:04.702756 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:10:04.702768 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:10:04.703430 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:10:04.703435 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:10:04.703568 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:10:04.704122 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:10:04.704139 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:10:04.707513 (+ 3374us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.34.254:33193: Server oryx/127.12.34.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":321,"threads_started":1}
W20260926 15:10:04.707966 21684 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:10:04.698802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.195:34609 (local address 127.12.34.254:33193)
0926 15:10:04.698977 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0926 15:10:04.698982 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:10:04.699394 (+ 412us) server_negotiation.cc:407] Connection header received
0926 15:10:04.699449 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:10:04.699452 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:10:04.699512 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:10:04.699598 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:10:04.700547 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:10:04.701653 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:10:04.702884 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:10:04.703293 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:10:04.703392 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:10:04.707730 (+ 4338us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.34.195:34609: BlockingRecv error: recv got EOF from 127.12.34.195:34609 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 26 15:10:05 dist-test-slave-865v krb5kdc[20681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.34.194@KRBTEST.COM for oryx/127.12.34.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 15:10:05 dist-test-slave-865v krb5kdc[20681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.34.194@KRBTEST.COM for oryx/127.12.34.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 15:10:05.029049 21715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:10:05.019818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:33193 (local address 127.12.34.194:37661)
0926 15:10:05.020301 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0926 15:10:05.020318 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 15:10:05.020432 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:10:05.020757 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:10:05.020761 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:10:05.021186 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:10:05.021380 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:10:05.021394 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:10:05.022537 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:10:05.022541 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:10:05.023870 (+ 1329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:10:05.023881 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:10:05.024721 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:10:05.024727 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:10:05.024914 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:10:05.025524 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:10:05.025540 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:10:05.028686 (+ 3146us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.34.254:33193: Server oryx/127.12.34.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":245,"threads_started":1}
W20260926 15:10:05.029125 21684 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:10:05.020031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.194:37661 (local address 127.12.34.254:33193)
0926 15:10:05.020245 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0926 15:10:05.020251 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:10:05.020393 (+ 142us) server_negotiation.cc:407] Connection header received
0926 15:10:05.020537 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:10:05.020541 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:10:05.020599 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:10:05.020677 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:10:05.021534 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:10:05.022387 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:10:05.024007 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:10:05.024520 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:10:05.024679 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:10:05.028855 (+ 4176us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.34.194:37661: BlockingRecv error: recv got EOF from 127.12.34.194:37661 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 26 15:10:05 dist-test-slave-865v krb5kdc[20681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.34.193@KRBTEST.COM for oryx/127.12.34.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 15:10:05 dist-test-slave-865v krb5kdc[20681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.34.193@KRBTEST.COM for oryx/127.12.34.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 15:10:05.408499 21716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:10:05.389475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:33193 (local address 127.12.34.193:39145)
0926 15:10:05.390006 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0926 15:10:05.390021 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 15:10:05.390177 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:10:05.390551 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:10:05.390563 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:10:05.391121 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:10:05.391422 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:10:05.391436 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:10:05.392414 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:10:05.392419 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:10:05.393470 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:10:05.393482 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:10:05.394184 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:10:05.394189 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:10:05.394320 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:10:05.404754 (+ 10434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:10:05.404784 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:10:05.408279 (+ 3495us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.34.254:33193: Server oryx/127.12.34.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":299,"threads_started":1}
W20260926 15:10:05.408707 21684 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:10:05.389661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.193:39145 (local address 127.12.34.254:33193)
0926 15:10:05.389822 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0926 15:10:05.389826 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:10:05.390128 (+ 302us) server_negotiation.cc:407] Connection header received
0926 15:10:05.390306 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:10:05.390311 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:10:05.390391 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:10:05.390488 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:10:05.391550 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:10:05.392254 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:10:05.393603 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:10:05.393996 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:10:05.394101 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:10:05.408471 (+ 14370us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.34.193:39145: BlockingRecv error: recv got EOF from 127.12.34.193:39145 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 26 15:10:05 dist-test-slave-865v krb5kdc[20681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.34.195@KRBTEST.COM for oryx/127.12.34.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 15:10:05 dist-test-slave-865v krb5kdc[20681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.34.195@KRBTEST.COM for oryx/127.12.34.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 15:10:05.718164 21717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:10:05.708910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:33193 (local address 127.12.34.195:58399)
0926 15:10:05.709467 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0926 15:10:05.709483 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 15:10:05.709595 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:10:05.709902 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:10:05.709905 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:10:05.710305 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:10:05.710518 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:10:05.710529 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:10:05.711509 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:10:05.711513 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:10:05.712614 (+ 1101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:10:05.712645 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:10:05.713613 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:10:05.713619 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:10:05.713795 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:10:05.714434 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:10:05.714451 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:10:05.717914 (+ 3463us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.34.254:33193: Server oryx/127.12.34.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":308,"threads_started":1}
W20260926 15:10:05.718475 21684 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:10:05.708970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.195:58399 (local address 127.12.34.254:33193)
0926 15:10:05.709223 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0926 15:10:05.709233 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 15:10:05.709562 (+ 329us) server_negotiation.cc:407] Connection header received
0926 15:10:05.709713 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:10:05.709717 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:10:05.709778 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:10:05.709861 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:10:05.710647 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:10:05.711381 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:10:05.712889 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:10:05.713456 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:10:05.713598 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:10:05.718136 (+ 4538us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.34.195:58399: BlockingRecv error: recv got EOF from 127.12.34.195:58399 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 26 15:10:06 dist-test-slave-865v krb5kdc[20681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.34.194@KRBTEST.COM for oryx/127.12.34.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 15:10:06 dist-test-slave-865v krb5kdc[20681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.34.194@KRBTEST.COM for oryx/127.12.34.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 15:10:06.040081 21718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:10:06.030370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:33193 (local address 127.12.34.194:40741)
0926 15:10:06.030978 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0926 15:10:06.030994 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 15:10:06.031160 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:10:06.031496 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:10:06.031501 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:10:06.031936 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:10:06.032102 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:10:06.032110 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:10:06.033148 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:10:06.033156 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:10:06.034113 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:10:06.034126 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:10:06.035202 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:10:06.035209 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:10:06.035392 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:10:06.036190 (+ 798us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:10:06.036211 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:10:06.039868 (+ 3657us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.34.254:33193: Server oryx/127.12.34.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":361,"threads_started":1}
W20260926 15:10:06.040446 21684 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:10:06.030569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.194:40741 (local address 127.12.34.254:33193)
0926 15:10:06.030845 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0926 15:10:06.030851 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:10:06.031079 (+ 228us) server_negotiation.cc:407] Connection header received
0926 15:10:06.031290 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:10:06.031297 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:10:06.031361 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:10:06.031471 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:10:06.032241 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:10:06.032989 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:10:06.034416 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:10:06.035047 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:10:06.035147 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:10:06.040152 (+ 5005us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.34.194:40741: BlockingRecv error: recv got EOF from 127.12.34.194:40741 (error 108)
Metrics: {"server-negotiator.queue_time_us":109}
Sep 26 15:10:06 dist-test-slave-865v krb5kdc[20681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.34.193@KRBTEST.COM for oryx/127.12.34.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 15:10:06 dist-test-slave-865v krb5kdc[20681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.34.193@KRBTEST.COM for oryx/127.12.34.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 15:10:06.418696 21719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:10:06.409563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:33193 (local address 127.12.34.193:41263)
0926 15:10:06.410120 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0926 15:10:06.410136 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 15:10:06.410256 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:10:06.410559 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:10:06.410562 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:10:06.410954 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:10:06.411116 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:10:06.411124 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:10:06.412280 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:10:06.412284 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:10:06.413268 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:10:06.413279 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:10:06.413908 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:10:06.413913 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:10:06.414050 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:10:06.414708 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:10:06.414726 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:10:06.418467 (+ 3741us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.34.254:33193: Server oryx/127.12.34.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":316,"threads_started":1}
W20260926 15:10:06.418865 21684 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:10:06.409613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.193:41263 (local address 127.12.34.254:33193)
0926 15:10:06.409789 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0926 15:10:06.409793 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:10:06.410245 (+ 452us) server_negotiation.cc:407] Connection header received
0926 15:10:06.410367 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:10:06.410371 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:10:06.410433 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:10:06.410519 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:10:06.411250 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:10:06.412167 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:10:06.413390 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:10:06.413767 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:10:06.413866 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:10:06.418635 (+ 4769us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.34.193:41263: BlockingRecv error: recv got EOF from 127.12.34.193:41263 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 26 15:10:06 dist-test-slave-865v krb5kdc[20681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.34.195@KRBTEST.COM for oryx/127.12.34.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 15:10:06 dist-test-slave-865v krb5kdc[20681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.34.195@KRBTEST.COM for oryx/127.12.34.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 15:10:06.729769 21720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:10:06.719449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:33193 (local address 127.12.34.195:57897)
0926 15:10:06.720028 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0926 15:10:06.720044 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 15:10:06.720163 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:10:06.720596 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:10:06.720600 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:10:06.720995 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:10:06.721210 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:10:06.721227 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:10:06.722381 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:10:06.722385 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:10:06.723418 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:10:06.723430 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:10:06.724310 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:10:06.724315 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:10:06.724519 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:10:06.725245 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:10:06.725292 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:10:06.729543 (+ 4251us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.34.254:33193: Server oryx/127.12.34.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":363,"threads_started":1}
W20260926 15:10:06.729990 21684 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:10:06.719529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.195:57897 (local address 127.12.34.254:33193)
0926 15:10:06.719714 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0926 15:10:06.719721 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:10:06.720157 (+ 436us) server_negotiation.cc:407] Connection header received
0926 15:10:06.720375 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:10:06.720388 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:10:06.720453 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:10:06.720535 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:10:06.721351 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:10:06.722191 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:10:06.723585 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:10:06.724046 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:10:06.724278 (+ 232us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:10:06.729742 (+ 5464us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.34.195:57897: BlockingRecv error: recv got EOF from 127.12.34.195:57897 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 26 15:10:07 dist-test-slave-865v krb5kdc[20681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.34.194@KRBTEST.COM for oryx/127.12.34.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 15:10:07 dist-test-slave-865v krb5kdc[20681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.34.194@KRBTEST.COM for oryx/127.12.34.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 15:10:07.050729 21721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:10:07.041150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:33193 (local address 127.12.34.194:54913)
0926 15:10:07.041682 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0926 15:10:07.041698 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 15:10:07.041853 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:10:07.042202 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:10:07.042206 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:10:07.042624 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:10:07.042791 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:10:07.042800 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:10:07.043825 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:10:07.043833 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:10:07.044759 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:10:07.044770 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:10:07.045564 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:10:07.045570 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:10:07.045749 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:10:07.046413 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:10:07.046431 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:10:07.050498 (+ 4067us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.34.254:33193: Server oryx/127.12.34.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":330,"threads_started":1}
W20260926 15:10:07.051059 21684 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:10:07.041195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.194:54913 (local address 127.12.34.254:33193)
0926 15:10:07.041432 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0926 15:10:07.041440 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 15:10:07.041793 (+ 353us) server_negotiation.cc:407] Connection header received
0926 15:10:07.041987 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:10:07.041993 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:10:07.042067 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:10:07.042171 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:10:07.042941 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:10:07.043688 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:10:07.044898 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:10:07.045422 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:10:07.045553 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:10:07.050741 (+ 5188us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.34.194:54913: BlockingRecv error: recv got EOF from 127.12.34.194:54913 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 26 15:10:07 dist-test-slave-865v krb5kdc[20681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.34.193@KRBTEST.COM for oryx/127.12.34.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 15:10:07 dist-test-slave-865v krb5kdc[20681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.34.193@KRBTEST.COM for oryx/127.12.34.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 15:10:07.429375 21722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:10:07.419847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:33193 (local address 127.12.34.193:59505)
0926 15:10:07.420381 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0926 15:10:07.420405 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 15:10:07.420568 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:10:07.421012 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:10:07.421017 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:10:07.421638 (+ 621us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:10:07.421811 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:10:07.421820 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:10:07.422907 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:10:07.422912 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:10:07.423911 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:10:07.423924 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:10:07.424763 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:10:07.424768 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:10:07.424905 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:10:07.425535 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:10:07.425553 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:10:07.429130 (+ 3577us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.34.254:33193: Server oryx/127.12.34.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":256,"threads_started":1}
W20260926 15:10:07.429630 21684 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:10:07.419920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.193:59505 (local address 127.12.34.254:33193)
0926 15:10:07.420135 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0926 15:10:07.420144 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 15:10:07.420526 (+ 382us) server_negotiation.cc:407] Connection header received
0926 15:10:07.420748 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:10:07.420752 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:10:07.420833 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:10:07.420932 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:10:07.421945 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:10:07.422772 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:10:07.424047 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:10:07.424629 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:10:07.424748 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:10:07.429390 (+ 4642us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.34.193:59505: BlockingRecv error: recv got EOF from 127.12.34.193:59505 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 26 15:10:07 dist-test-slave-865v krb5kdc[20681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.34.195@KRBTEST.COM for oryx/127.12.34.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 15:10:07 dist-test-slave-865v krb5kdc[20681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.34.195@KRBTEST.COM for oryx/127.12.34.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 15:10:07.739886 21723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:10:07.730749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:33193 (local address 127.12.34.195:52239)
0926 15:10:07.731340 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0926 15:10:07.731375 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 15:10:07.731487 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:10:07.731844 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:10:07.731849 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:10:07.732446 (+ 597us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:10:07.732680 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:10:07.732692 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:10:07.733875 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:10:07.733879 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:10:07.734844 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:10:07.734856 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:10:07.735512 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:10:07.735516 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:10:07.735712 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:10:07.736440 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:10:07.736456 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:10:07.739641 (+ 3185us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.34.254:33193: Server oryx/127.12.34.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":338,"threads_started":1}
W20260926 15:10:07.740118 21684 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:10:07.730828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.195:52239 (local address 127.12.34.254:33193)
0926 15:10:07.730995 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0926 15:10:07.730999 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:10:07.731473 (+ 474us) server_negotiation.cc:407] Connection header received
0926 15:10:07.731600 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:10:07.731605 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:10:07.731679 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:10:07.731806 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:10:07.732825 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:10:07.733746 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:10:07.734962 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:10:07.735362 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:10:07.735458 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:10:07.739855 (+ 4397us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.34.195:52239: BlockingRecv error: recv got EOF from 127.12.34.195:52239 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 26 15:10:08 dist-test-slave-865v krb5kdc[20681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.34.194@KRBTEST.COM for oryx/127.12.34.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 15:10:08 dist-test-slave-865v krb5kdc[20681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.34.194@KRBTEST.COM for oryx/127.12.34.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 15:10:08.062507 21724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:10:08.051802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:33193 (local address 127.12.34.194:52851)
0926 15:10:08.052549 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0926 15:10:08.052570 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 15:10:08.052800 (+ 230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:10:08.053354 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:10:08.053360 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:10:08.053822 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:10:08.053972 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:10:08.053980 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:10:08.055392 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:10:08.055397 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:10:08.056388 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:10:08.056400 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:10:08.057183 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:10:08.057190 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:10:08.057330 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:10:08.058084 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:10:08.058111 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:10:08.062257 (+ 4146us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.34.254:33193: Server oryx/127.12.34.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":573,"thread_start_us":412,"threads_started":1}
W20260926 15:10:08.062687 21684 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:10:08.052017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.194:52851 (local address 127.12.34.254:33193)
0926 15:10:08.052383 (+ 366us) server_negotiation.cc:206] Beginning negotiation
0926 15:10:08.052391 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 15:10:08.052699 (+ 308us) server_negotiation.cc:407] Connection header received
0926 15:10:08.053041 (+ 342us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:10:08.053045 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:10:08.053169 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:10:08.053303 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:10:08.054179 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:10:08.055245 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:10:08.056516 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:10:08.056945 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:10:08.057040 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:10:08.062457 (+ 5417us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.34.194:52851: BlockingRecv error: recv got EOF from 127.12.34.194:52851 (error 108)
Metrics: {"server-negotiator.queue_time_us":116}
Sep 26 15:10:08 dist-test-slave-865v krb5kdc[20681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.34.193@KRBTEST.COM for oryx/127.12.34.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 15:10:08 dist-test-slave-865v krb5kdc[20681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.34.193@KRBTEST.COM for oryx/127.12.34.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 15:10:08.439991 21725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:10:08.430843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:33193 (local address 127.12.34.193:56761)
0926 15:10:08.431341 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0926 15:10:08.431356 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 15:10:08.431508 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:10:08.431843 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:10:08.431847 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:10:08.432225 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:10:08.432402 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:10:08.432410 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:10:08.433516 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:10:08.433522 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:10:08.434428 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:10:08.434438 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:10:08.435167 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:10:08.435173 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:10:08.435355 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:10:08.435999 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:10:08.436016 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:10:08.439558 (+ 3542us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.34.254:33193: Server oryx/127.12.34.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":270,"threads_started":1}
W20260926 15:10:08.440191 21684 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:10:08.430913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.193:56761 (local address 127.12.34.254:33193)
0926 15:10:08.431199 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0926 15:10:08.431206 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:10:08.431437 (+ 231us) server_negotiation.cc:407] Connection header received
0926 15:10:08.431634 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:10:08.431639 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:10:08.431715 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:10:08.431812 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:10:08.432533 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:10:08.433385 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:10:08.434562 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:10:08.435010 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:10:08.435150 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:10:08.439880 (+ 4730us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.34.193:56761: BlockingRecv error: recv got EOF from 127.12.34.193:56761 (error 108)
Metrics: {"server-negotiator.queue_time_us":109}
Sep 26 15:10:08 dist-test-slave-865v krb5kdc[20681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.34.195@KRBTEST.COM for oryx/127.12.34.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 15:10:08 dist-test-slave-865v krb5kdc[20681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.34.195@KRBTEST.COM for oryx/127.12.34.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 15:10:08.750609 21726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:10:08.740959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:33193 (local address 127.12.34.195:51313)
0926 15:10:08.741493 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0926 15:10:08.741508 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 15:10:08.741655 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:10:08.742352 (+ 697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:10:08.742358 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:10:08.742866 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:10:08.743080 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:10:08.743093 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:10:08.744569 (+ 1476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:10:08.744573 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:10:08.745518 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:10:08.745529 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:10:08.746210 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:10:08.746214 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:10:08.746333 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:10:08.746979 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:10:08.747002 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:10:08.750393 (+ 3391us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.34.254:33193: Server oryx/127.12.34.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":317,"threads_started":1}
W20260926 15:10:08.750880 21684 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:10:08.741008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.195:51313 (local address 127.12.34.254:33193)
0926 15:10:08.741299 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0926 15:10:08.741304 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:10:08.741586 (+ 282us) server_negotiation.cc:407] Connection header received
0926 15:10:08.741821 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:10:08.741826 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:10:08.741921 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:10:08.742046 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:10:08.743381 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:10:08.744432 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:10:08.745672 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:10:08.746056 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:10:08.746170 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:10:08.750615 (+ 4445us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.34.195:51313: BlockingRecv error: recv got EOF from 127.12.34.195:51313 (error 108)
Metrics: {"server-negotiator.queue_time_us":166}
Sep 26 15:10:09 dist-test-slave-865v krb5kdc[20681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.34.194@KRBTEST.COM for oryx/127.12.34.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 15:10:09 dist-test-slave-865v krb5kdc[20681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.34.194@KRBTEST.COM for oryx/127.12.34.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 15:10:09.073042 21727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:10:09.063659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:33193 (local address 127.12.34.194:59265)
0926 15:10:09.064161 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0926 15:10:09.064178 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 15:10:09.064328 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:10:09.064657 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:10:09.064661 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:10:09.065127 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:10:09.065319 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:10:09.065328 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:10:09.066302 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:10:09.066310 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:10:09.067379 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:10:09.067390 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:10:09.068257 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:10:09.068264 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:10:09.068513 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:10:09.069244 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:10:09.069264 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:10:09.072793 (+ 3529us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.34.254:33193: Server oryx/127.12.34.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":272,"threads_started":1}
W20260926 15:10:09.073387 21684 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:10:09.063813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.194:59265 (local address 127.12.34.254:33193)
0926 15:10:09.064048 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0926 15:10:09.064054 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:10:09.064301 (+ 247us) server_negotiation.cc:407] Connection header received
0926 15:10:09.064461 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:10:09.064466 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:10:09.064533 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:10:09.064634 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:10:09.065459 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:10:09.066165 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:10:09.067511 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:10:09.067991 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:10:09.068119 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:10:09.073011 (+ 4892us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.34.194:59265: BlockingRecv error: recv got EOF from 127.12.34.194:59265 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Sep 26 15:10:09 dist-test-slave-865v krb5kdc[20681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.34.193@KRBTEST.COM for oryx/127.12.34.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 15:10:09 dist-test-slave-865v krb5kdc[20681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.34.193@KRBTEST.COM for oryx/127.12.34.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 15:10:09.450533 21728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:10:09.441337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:33193 (local address 127.12.34.193:33625)
0926 15:10:09.441897 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0926 15:10:09.441913 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 15:10:09.442073 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:10:09.442415 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:10:09.442420 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:10:09.442847 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:10:09.443005 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:10:09.443013 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:10:09.444012 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:10:09.444020 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:10:09.444970 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:10:09.444981 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:10:09.445699 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:10:09.445705 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:10:09.445873 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:10:09.446557 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:10:09.446590 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:10:09.450305 (+ 3715us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.34.254:33193: Server oryx/127.12.34.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":292,"threads_started":1}
W20260926 15:10:09.450858 21684 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:10:09.441423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.193:33625 (local address 127.12.34.254:33193)
0926 15:10:09.441681 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0926 15:10:09.441689 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 15:10:09.442006 (+ 317us) server_negotiation.cc:407] Connection header received
0926 15:10:09.442219 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:10:09.442223 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:10:09.442291 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:10:09.442389 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:10:09.443144 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:10:09.443879 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:10:09.445132 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:10:09.445551 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:10:09.445693 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:10:09.450497 (+ 4804us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.34.193:33625: BlockingRecv error: recv got EOF from 127.12.34.193:33625 (error 108)
Metrics: {"server-negotiator.queue_time_us":104}
I20260926 15:10:09.482757 12427 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 21268
I20260926 15:10:09.496592 21730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:10:09.483984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37028 (local address 127.12.34.193:39325)
0926 15:10:09.484675 (+ 691us) server_negotiation.cc:206] Beginning negotiation
0926 15:10:09.484682 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:10:09.484712 (+ 30us) server_negotiation.cc:407] Connection header received
0926 15:10:09.484773 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:10:09.484779 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:10:09.485026 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:10:09.485271 (+ 245us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:10:09.486660 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:10:09.487782 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:10:09.489166 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:10:09.489709 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:10:09.489816 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:10:09.492376 (+ 2560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:10:09.492385 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:10:09.492398 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:10:09.492439 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:10:09.495770 (+ 3331us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.12.34.193@KRBTEST.COM not found in keytab (ticket kvno 2)
0926 15:10:09.496185 (+ 415us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:37028: Request ticket server kudu/127.12.34.193@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":504,"thread_start_us":373,"threads_started":1}
W20260926 15:10:09.496635 12427 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 21268: Not authorized: Client connection negotiation failed: client connection to 127.12.34.193:39325: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.12.34.193@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260926 15:10:09.496759 21730 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:37028: Request ticket server kudu/127.12.34.193@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260926 15:10:09.497041 12427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8uoJGv/build/asan/bin/kudu with pid 21268
I20260926 15:10:09.515758 12427 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 21419
I20260926 15:10:09.528523 21731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:10:09.516955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52084 (local address 127.12.34.194:43743)
0926 15:10:09.517589 (+ 634us) server_negotiation.cc:206] Beginning negotiation
0926 15:10:09.517597 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 15:10:09.517625 (+ 28us) server_negotiation.cc:407] Connection header received
0926 15:10:09.517691 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:10:09.517700 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:10:09.517928 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:10:09.518133 (+ 205us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:10:09.519421 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:10:09.520462 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:10:09.521824 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:10:09.522388 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:10:09.522530 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:10:09.525033 (+ 2503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:10:09.525061 (+ 28us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:10:09.525128 (+ 67us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:10:09.525195 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:10:09.527965 (+ 2770us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.12.34.194@KRBTEST.COM not found in keytab (ticket kvno 2)
0926 15:10:09.528268 (+ 303us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:52084: Request ticket server kudu/127.12.34.194@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":458,"thread_start_us":320,"threads_started":1}
W20260926 15:10:09.528627 12427 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 21419: Not authorized: Client connection negotiation failed: client connection to 127.12.34.194:43743: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.12.34.194@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260926 15:10:09.528676 21731 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:52084: Request ticket server kudu/127.12.34.194@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260926 15:10:09.528784 12427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8uoJGv/build/asan/bin/kudu with pid 21419
I20260926 15:10:09.547065 12427 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 21552
I20260926 15:10:09.559469 21732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:10:09.548218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36980 (local address 127.12.34.195:41957)
0926 15:10:09.548744 (+ 526us) server_negotiation.cc:206] Beginning negotiation
0926 15:10:09.548758 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 15:10:09.548783 (+ 25us) server_negotiation.cc:407] Connection header received
0926 15:10:09.548845 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:10:09.548852 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:10:09.549142 (+ 290us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:10:09.549355 (+ 213us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:10:09.550729 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:10:09.551714 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:10:09.552879 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:10:09.553610 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:10:09.553761 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:10:09.556124 (+ 2363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:10:09.556133 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:10:09.556146 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:10:09.556198 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:10:09.558788 (+ 2590us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.12.34.195@KRBTEST.COM not found in keytab (ticket kvno 2)
0926 15:10:09.559127 (+ 339us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:36980: Request ticket server kudu/127.12.34.195@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":244,"threads_started":1}
W20260926 15:10:09.559499 12427 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 21552: Not authorized: Client connection negotiation failed: client connection to 127.12.34.195:41957: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.12.34.195@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260926 15:10:09.559574 21732 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:36980: Request ticket server kudu/127.12.34.195@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260926 15:10:09.559645 12427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8uoJGv/build/asan/bin/kudu with pid 21552
I20260926 15:10:09.576193 12427 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 21199
I20260926 15:10:09.577785 21261 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:10:09.693069 12427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8uoJGv/build/asan/bin/kudu with pid 21199
2026-09-26T15:10:09Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (21426 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-26T15:10:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T15:10:09Z Disabled control of system clock
I20260926 15:10:09.769194 12427 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8uoJGv/build/asan/bin/kudu
/tmp/dist-test-task8uoJGv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790435318278681-12427-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790435318278681-12427-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790435318278681-12427-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.34.254:33097
--webserver_interface=127.12.34.254
--webserver_port=0
--builtin_ntp_servers=127.12.34.212:38875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.34.254:33097
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260926 15:10:10.099478 21737 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:10:10.099792 21737 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:10:10.099874 21737 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:10:10.107722 21737 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:10:10.107908 21737 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:10:10.108054 21737 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:10:10.108168 21737 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:10:10.108225 21737 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:10:10.119845 21737 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.34.212:38875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790435318278681-12427-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790435318278681-12427-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.34.254:33097
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.34.254:33097
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.34.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790435318278681-12427-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:10:10.121248 21737 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:10:10.123523 21737 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:10:10.135963 21742 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:10:10.136152 21743 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:10:10.136197 21745 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:10:10.137436 21737 server_base.cc:1034] running on GCE node
I20260926 15:10:10.138283 21737 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:10:10.139901 21737 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:10:10.141403 21737 hybrid_clock.cc:648] HybridClock initialized: now 1790435410141321 us; error 45 us; skew 500 ppm
I20260926 15:10:10.142017 21737 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 15:10:10.145007 21737 webserver.cc:460] Webserver started at http://127.12.34.254:34077/ using document root <none> and password file <none>
I20260926 15:10:10.145828 21737 fs_manager.cc:359] Metadata directory not provided
I20260926 15:10:10.145972 21737 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:10:10.146301 21737 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:10:10.149282 21737 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790435318278681-12427-0/minicluster-data/master-0/data/instance:
uuid: "18fa8eb1ddce4c9ea3669b0361ae118d"
format_stamp: "Formatted at 2026-09-26 15:10:10 on dist-test-slave-865v"
server_key: "474047c758ada13bf5dfe47caed6c86b"
server_key_iv: "86fdf3315fac11b4b7787de6c6d4cb25"
server_key_version: "encryptionkey@0"
I20260926 15:10:10.150071 21737 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790435318278681-12427-0/minicluster-data/master-0/wal/instance:
uuid: "18fa8eb1ddce4c9ea3669b0361ae118d"
format_stamp: "Formatted at 2026-09-26 15:10:10 on dist-test-slave-865v"
server_key: "474047c758ada13bf5dfe47caed6c86b"
server_key_iv: "86fdf3315fac11b4b7787de6c6d4cb25"
server_key_version: "encryptionkey@0"
I20260926 15:10:10.155197 21737 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.005s
I20260926 15:10:10.158601 21751 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:10:10.160194 21737 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 15:10:10.160385 21737 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790435318278681-12427-0/minicluster-data/master-0/data,/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790435318278681-12427-0/minicluster-data/master-0/wal
uuid: "18fa8eb1ddce4c9ea3669b0361ae118d"
format_stamp: "Formatted at 2026-09-26 15:10:10 on dist-test-slave-865v"
server_key: "474047c758ada13bf5dfe47caed6c86b"
server_key_iv: "86fdf3315fac11b4b7787de6c6d4cb25"
server_key_version: "encryptionkey@0"
I20260926 15:10:10.160597 21737 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790435318278681-12427-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790435318278681-12427-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790435318278681-12427-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:10:10.175901 21737 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:10:10.176766 21737 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:10:10.177055 21737 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:10:10.195261 21737 rpc_server.cc:307] RPC server started. Bound to: 127.12.34.254:33097
I20260926 15:10:10.195268 21802 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.34.254:33097 every 8 connection(s)
I20260926 15:10:10.196568 21737 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
I20260926 15:10:10.200210 21803 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:10:10.200630 12427 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8uoJGv/build/asan/bin/kudu as pid 21737
I20260926 15:10:10.200836 12427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790435318278681-12427-0/minicluster-data/master-0/wal/instance
I20260926 15:10:10.201532 12427 external_mini_cluster.cc:1428] Setting key 6d6a6ded72878b11dff5ce5684fce241
I20260926 15:10:10.212477 21803 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 18fa8eb1ddce4c9ea3669b0361ae118d: Bootstrap starting.
I20260926 15:10:10.214561 21806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:10:10.203741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60802 (local address 127.12.34.254:33097)
0926 15:10:10.204670 (+ 929us) server_negotiation.cc:206] Beginning negotiation
0926 15:10:10.204680 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 15:10:10.204717 (+ 37us) server_negotiation.cc:407] Connection header received
0926 15:10:10.206498 (+ 1781us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:10:10.206517 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:10:10.206938 (+ 421us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:10:10.207334 (+ 396us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:10:10.207908 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:10:10.209066 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:10:10.210398 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:10:10.211036 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:10:10.211239 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:10:10.212382 (+ 1143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:10:10.212389 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:10:10.212402 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:10:10.212943 (+ 541us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:10:10.213010 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:10:10.213030 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:10:10.213225 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0926 15:10:10.213436 (+ 211us) server_negotiation.cc:299] Negotiation successful
0926 15:10:10.213702 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":548,"thread_start_us":356,"threads_started":1}
I20260926 15:10:10.215855 21803 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 18fa8eb1ddce4c9ea3669b0361ae118d: Neither blocks nor log segments found. Creating new log.
I20260926 15:10:10.216856 21803 log.cc:824] T 00000000000000000000000000000000 P 18fa8eb1ddce4c9ea3669b0361ae118d: Log is configured to *not* fsync() on all Append() calls
I20260926 15:10:10.219806 21803 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 18fa8eb1ddce4c9ea3669b0361ae118d: No bootstrap required, opened a new log
I20260926 15:10:10.227850 21803 raft_consensus.cc:348] T 00000000000000000000000000000000 P 18fa8eb1ddce4c9ea3669b0361ae118d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18fa8eb1ddce4c9ea3669b0361ae118d" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 33097 } }
I20260926 15:10:10.228257 21803 raft_consensus.cc:374] T 00000000000000000000000000000000 P 18fa8eb1ddce4c9ea3669b0361ae118d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:10:10.228366 21803 raft_consensus.cc:729] T 00000000000000000000000000000000 P 18fa8eb1ddce4c9ea3669b0361ae118d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18fa8eb1ddce4c9ea3669b0361ae118d, State: Initialized, Role: FOLLOWER
I20260926 15:10:10.228957 21803 consensus_queue.cc:260] T 00000000000000000000000000000000 P 18fa8eb1ddce4c9ea3669b0361ae118d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18fa8eb1ddce4c9ea3669b0361ae118d" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 33097 } }
I20260926 15:10:10.229245 21803 raft_consensus.cc:388] T 00000000000000000000000000000000 P 18fa8eb1ddce4c9ea3669b0361ae118d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 15:10:10.229420 21803 raft_consensus.cc:482] T 00000000000000000000000000000000 P 18fa8eb1ddce4c9ea3669b0361ae118d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 15:10:10.229596 21803 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 18fa8eb1ddce4c9ea3669b0361ae118d [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:10:10.231671 21803 raft_consensus.cc:504] T 00000000000000000000000000000000 P 18fa8eb1ddce4c9ea3669b0361ae118d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18fa8eb1ddce4c9ea3669b0361ae118d" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 33097 } }
I20260926 15:10:10.232266 21803 leader_election.cc:302] T 00000000000000000000000000000000 P 18fa8eb1ddce4c9ea3669b0361ae118d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 18fa8eb1ddce4c9ea3669b0361ae118d; no voters:
I20260926 15:10:10.232890 21803 leader_election.cc:288] T 00000000000000000000000000000000 P 18fa8eb1ddce4c9ea3669b0361ae118d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 15:10:10.233054 21808 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 18fa8eb1ddce4c9ea3669b0361ae118d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:10:10.233807 21808 raft_consensus.cc:686] T 00000000000000000000000000000000 P 18fa8eb1ddce4c9ea3669b0361ae118d [term 1 LEADER]: Becoming Leader. State: Replica: 18fa8eb1ddce4c9ea3669b0361ae118d, State: Running, Role: LEADER
I20260926 15:10:10.234337 21808 consensus_queue.cc:237] T 00000000000000000000000000000000 P 18fa8eb1ddce4c9ea3669b0361ae118d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18fa8eb1ddce4c9ea3669b0361ae118d" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 33097 } }
I20260926 15:10:10.234934 21803 sys_catalog.cc:564] T 00000000000000000000000000000000 P 18fa8eb1ddce4c9ea3669b0361ae118d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:10:10.236938 21809 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18fa8eb1ddce4c9ea3669b0361ae118d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "18fa8eb1ddce4c9ea3669b0361ae118d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18fa8eb1ddce4c9ea3669b0361ae118d" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 33097 } } }
I20260926 15:10:10.237483 21809 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18fa8eb1ddce4c9ea3669b0361ae118d [sys.catalog]: This master's current role is: LEADER
I20260926 15:10:10.238193 21810 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18fa8eb1ddce4c9ea3669b0361ae118d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 18fa8eb1ddce4c9ea3669b0361ae118d. Latest consensus state: current_term: 1 leader_uuid: "18fa8eb1ddce4c9ea3669b0361ae118d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18fa8eb1ddce4c9ea3669b0361ae118d" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 33097 } } }
I20260926 15:10:10.238692 21810 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18fa8eb1ddce4c9ea3669b0361ae118d [sys.catalog]: This master's current role is: LEADER
I20260926 15:10:10.239974 21817 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:10:10.245007 21817 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 15:10:10.255965 21817 catalog_manager.cc:1347] Generated new cluster ID: ef20f171c1564c00893d1ab972ed5b60
I20260926 15:10:10.256083 21817 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 15:10:10.284849 21817 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 15:10:10.286520 21817 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:10:10.297258 21817 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 18fa8eb1ddce4c9ea3669b0361ae118d: Generated new TSK 0
I20260926 15:10:10.298312 21817 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 15:10:10.304589 12427 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8uoJGv/build/asan/bin/kudu
/tmp/dist-test-task8uoJGv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790435318278681-12427-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790435318278681-12427-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790435318278681-12427-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790435318278681-12427-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.34.193:0
--local_ip_for_outbound_sockets=127.12.34.193
--webserver_interface=127.12.34.193
--webserver_port=0
--tserver_master_addrs=127.12.34.254:33097
--builtin_ntp_servers=127.12.34.212:38875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260926 15:10:10.646509 21827 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:10:10.646809 21827 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:10:10.646965 21827 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:10:10.654147 21827 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:10:10.654281 21827 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:10:10.654453 21827 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.34.193
I20260926 15:10:10.663623 21827 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.34.212:38875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790435318278681-12427-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790435318278681-12427-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.34.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790435318278681-12427-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.34.193
--webserver_port=0
--tserver_master_addrs=127.12.34.254:33097
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.34.193
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790435318278681-12427-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:10:10.664822 21827 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:10:10.666819 21827 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:10:10.679204 21835 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:10:10.679239 21833 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:10:10.679720 21832 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:10:10.680397 21827 server_base.cc:1034] running on GCE node
I20260926 15:10:10.680996 21827 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:10:10.681969 21827 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:10:10.683346 21827 hybrid_clock.cc:648] HybridClock initialized: now 1790435410683250 us; error 51 us; skew 500 ppm
I20260926 15:10:10.683888 21827 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 15:10:10.692507 21827 webserver.cc:460] Webserver started at http://127.12.34.193:43229/ using document root <none> and password file <none>
I20260926 15:10:10.693368 21827 fs_manager.cc:359] Metadata directory not provided
I20260926 15:10:10.693506 21827 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:10:10.693857 21827 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:10:10.696765 21827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790435318278681-12427-0/minicluster-data/ts-0/data/instance:
uuid: "c08fe5a26b894cdf916eb9ee5ef9047f"
format_stamp: "Formatted at 2026-09-26 15:10:10 on dist-test-slave-865v"
server_key: "8aaf12aefe1f490e7ea1814ffa096fc3"
server_key_iv: "790226c380f0114f6c5db7620b1e33e1"
server_key_version: "encryptionkey@0"
I20260926 15:10:10.697546 21827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790435318278681-12427-0/minicluster-data/ts-0/wal/instance:
uuid: "c08fe5a26b894cdf916eb9ee5ef9047f"
format_stamp: "Formatted at 2026-09-26 15:10:10 on dist-test-slave-865v"
server_key: "8aaf12aefe1f490e7ea1814ffa096fc3"
server_key_iv: "790226c380f0114f6c5db7620b1e33e1"
server_key_version: "encryptionkey@0"
I20260926 15:10:10.702999 21827 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260926 15:10:10.706808 21841 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:10:10.708060 21827 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 15:10:10.708237 21827 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790435318278681-12427-0/minicluster-data/ts-0/data,/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790435318278681-12427-0/minicluster-data/ts-0/wal
uuid: "c08fe5a26b894cdf916eb9ee5ef9047f"
format_stamp: "Formatted at 2026-09-26 15:10:10 on dist-test-slave-865v"
server_key: "8aaf12aefe1f490e7ea1814ffa096fc3"
server_key_iv: "790226c380f0114f6c5db7620b1e33e1"
server_key_version: "encryptionkey@0"
I20260926 15:10:10.708463 21827 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790435318278681-12427-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790435318278681-12427-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790435318278681-12427-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:10:10.736531 21827 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:10:10.737457 21827 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:10:10.737813 21827 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:10:10.738884 21827 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:10:10.740280 21827 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:10:10.740372 21827 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:10:10.740514 21827 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:10:10.740607 21827 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:10:10.775044 21827 rpc_server.cc:307] RPC server started. Bound to: 127.12.34.193:37649
I20260926 15:10:10.775108 21953 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.34.193:37649 every 8 connection(s)
I20260926 15:10:10.776320 21827 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790435318278681-12427-0/minicluster-data/ts-0/data/info.pb
I20260926 15:10:10.781385 12427 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8uoJGv/build/asan/bin/kudu as pid 21827
I20260926 15:10:10.781746 12427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790435318278681-12427-0/minicluster-data/ts-0/wal/instance
I20260926 15:10:10.782510 12427 external_mini_cluster.cc:1428] Setting key a0853884d4356324548bab65d02345e9
I20260926 15:10:10.785477 12427 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8uoJGv/build/asan/bin/kudu
/tmp/dist-test-task8uoJGv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790435318278681-12427-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790435318278681-12427-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790435318278681-12427-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790435318278681-12427-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.34.194:0
--local_ip_for_outbound_sockets=127.12.34.194
--webserver_interface=127.12.34.194
--webserver_port=0
--tserver_master_addrs=127.12.34.254:33097
--builtin_ntp_servers=127.12.34.212:38875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260926 15:10:10.790849 21956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:10:10.779751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.193:55239 (local address 127.12.34.254:33097)
0926 15:10:10.780345 (+ 594us) server_negotiation.cc:206] Beginning negotiation
0926 15:10:10.780353 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 15:10:10.782103 (+ 1750us) server_negotiation.cc:407] Connection header received
0926 15:10:10.784330 (+ 2227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:10:10.784335 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:10:10.784429 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:10:10.784535 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:10:10.785736 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:10:10.786426 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:10:10.787962 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:10:10.788358 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:10:10.788490 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:10:10.790004 (+ 1514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:10:10.790009 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:10:10.790013 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:10:10.790029 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:10:10.790065 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:10:10.790076 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:10:10.790192 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0926 15:10:10.790417 (+ 225us) server_negotiation.cc:299] Negotiation successful
0926 15:10:10.790509 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":281,"threads_started":1}
I20260926 15:10:10.791900 21957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:10:10.780523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:33097 (local address 127.12.34.193:55239)
0926 15:10:10.781907 (+ 1384us) negotiation.cc:107] Waiting for socket to connect
0926 15:10:10.781965 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0926 15:10:10.784023 (+ 2058us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:10:10.784781 (+ 758us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:10:10.784789 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:10:10.784886 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:10:10.785513 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:10:10.785528 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:10:10.786588 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:10:10.786594 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:10:10.787812 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:10:10.787824 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:10:10.788595 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:10:10.788601 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:10:10.788837 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:10:10.789660 (+ 823us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 15:10:10.789663 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:10:10.789700 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 15:10:10.789792 (+ 92us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:10:10.790190 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:10:10.790200 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:10:10.790210 (+ 10us) client_negotiation.cc:769] Sending connection context
0926 15:10:10.790393 (+ 183us) client_negotiation.cc:240] Negotiation successful
0926 15:10:10.790811 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":733,"thread_start_us":447,"threads_started":1}
I20260926 15:10:10.793496 21954 heartbeater.cc:347] Connected to a master server at 127.12.34.254:33097
I20260926 15:10:10.793807 21954 heartbeater.cc:464] Registering TS with master...
I20260926 15:10:10.794674 21954 heartbeater.cc:511] Master 127.12.34.254:33097 requested a full tablet report, sending...
I20260926 15:10:10.796973 21768 ts_manager.cc:194] Registered new tserver with Master: c08fe5a26b894cdf916eb9ee5ef9047f (127.12.34.193:37649)
I20260926 15:10:10.798502 21768 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.34.193:55239
W20260926 15:10:11.120986 21958 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:10:11.121256 21958 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:10:11.121419 21958 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:10:11.128476 21958 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:10:11.128566 21958 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:10:11.128759 21958 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.34.194
I20260926 15:10:11.137899 21958 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.34.212:38875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790435318278681-12427-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790435318278681-12427-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.34.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790435318278681-12427-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.34.194
--webserver_port=0
--tserver_master_addrs=127.12.34.254:33097
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.34.194
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790435318278681-12427-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:10:11.139194 21958 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:10:11.140813 21958 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:10:11.153357 21963 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:10:11.153404 21964 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:10:11.153651 21966 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:10:11.154304 21958 server_base.cc:1034] running on GCE node
I20260926 15:10:11.154932 21958 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:10:11.155928 21958 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:10:11.157210 21958 hybrid_clock.cc:648] HybridClock initialized: now 1790435411157129 us; error 117 us; skew 500 ppm
I20260926 15:10:11.157536 21958 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 15:10:11.159718 21958 webserver.cc:460] Webserver started at http://127.12.34.194:43657/ using document root <none> and password file <none>
I20260926 15:10:11.160420 21958 fs_manager.cc:359] Metadata directory not provided
I20260926 15:10:11.160552 21958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:10:11.160868 21958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:10:11.163611 21958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790435318278681-12427-0/minicluster-data/ts-1/data/instance:
uuid: "6339ab94578644278e2d8dffa3505c0d"
format_stamp: "Formatted at 2026-09-26 15:10:11 on dist-test-slave-865v"
server_key: "d05d998a2173dd69eae421328d62adf4"
server_key_iv: "6a1836380cfd7bf490e48d64d2d788ef"
server_key_version: "encryptionkey@0"
I20260926 15:10:11.164402 21958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790435318278681-12427-0/minicluster-data/ts-1/wal/instance:
uuid: "6339ab94578644278e2d8dffa3505c0d"
format_stamp: "Formatted at 2026-09-26 15:10:11 on dist-test-slave-865v"
server_key: "d05d998a2173dd69eae421328d62adf4"
server_key_iv: "6a1836380cfd7bf490e48d64d2d788ef"
server_key_version: "encryptionkey@0"
I20260926 15:10:11.169405 21958 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.006s
I20260926 15:10:11.172549 21972 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:10:11.173880 21958 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 15:10:11.174049 21958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790435318278681-12427-0/minicluster-data/ts-1/data,/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790435318278681-12427-0/minicluster-data/ts-1/wal
uuid: "6339ab94578644278e2d8dffa3505c0d"
format_stamp: "Formatted at 2026-09-26 15:10:11 on dist-test-slave-865v"
server_key: "d05d998a2173dd69eae421328d62adf4"
server_key_iv: "6a1836380cfd7bf490e48d64d2d788ef"
server_key_version: "encryptionkey@0"
I20260926 15:10:11.174258 21958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790435318278681-12427-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790435318278681-12427-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790435318278681-12427-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:10:11.195783 21958 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:10:11.196693 21958 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:10:11.197003 21958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:10:11.197991 21958 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:10:11.199592 21958 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:10:11.199692 21958 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:10:11.199783 21958 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:10:11.199849 21958 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:10:11.233903 21958 rpc_server.cc:307] RPC server started. Bound to: 127.12.34.194:42887
I20260926 15:10:11.233958 22084 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.34.194:42887 every 8 connection(s)
I20260926 15:10:11.235136 21958 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790435318278681-12427-0/minicluster-data/ts-1/data/info.pb
I20260926 15:10:11.238353 12427 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8uoJGv/build/asan/bin/kudu as pid 21958
I20260926 15:10:11.238512 12427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790435318278681-12427-0/minicluster-data/ts-1/wal/instance
I20260926 15:10:11.239125 12427 external_mini_cluster.cc:1428] Setting key fa77b3a00b59f743c0ce0b18a74887de
I20260926 15:10:11.241817 12427 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8uoJGv/build/asan/bin/kudu
/tmp/dist-test-task8uoJGv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790435318278681-12427-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790435318278681-12427-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790435318278681-12427-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790435318278681-12427-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.34.195:0
--local_ip_for_outbound_sockets=127.12.34.195
--webserver_interface=127.12.34.195
--webserver_port=0
--tserver_master_addrs=127.12.34.254:33097
--builtin_ntp_servers=127.12.34.212:38875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260926 15:10:11.247668 21956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:10:11.237773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.194:60963 (local address 127.12.34.254:33097)
0926 15:10:11.237948 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0926 15:10:11.237952 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:10:11.239620 (+ 1668us) server_negotiation.cc:407] Connection header received
0926 15:10:11.241630 (+ 2010us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:10:11.241636 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:10:11.241717 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:10:11.241832 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:10:11.242789 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:10:11.243492 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:10:11.244961 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:10:11.245418 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:10:11.245545 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:10:11.246935 (+ 1390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:10:11.246944 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:10:11.246952 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:10:11.246976 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:10:11.247017 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:10:11.247036 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:10:11.247158 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0926 15:10:11.247392 (+ 234us) server_negotiation.cc:299] Negotiation successful
0926 15:10:11.247462 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 15:10:11.249032 22087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:10:11.238413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:33097 (local address 127.12.34.194:60963)
0926 15:10:11.239475 (+ 1062us) negotiation.cc:107] Waiting for socket to connect
0926 15:10:11.239518 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 15:10:11.241393 (+ 1875us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:10:11.242036 (+ 643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:10:11.242043 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:10:11.242123 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:10:11.242630 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:10:11.242644 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:10:11.243637 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:10:11.243644 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:10:11.244819 (+ 1175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:10:11.244830 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:10:11.245630 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:10:11.245638 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:10:11.245850 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:10:11.246543 (+ 693us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 15:10:11.246546 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:10:11.246585 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 15:10:11.246720 (+ 135us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:10:11.247162 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:10:11.247174 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:10:11.247182 (+ 8us) client_negotiation.cc:769] Sending connection context
0926 15:10:11.247375 (+ 193us) client_negotiation.cc:240] Negotiation successful
0926 15:10:11.247776 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":304,"threads_started":1}
I20260926 15:10:11.250363 22085 heartbeater.cc:347] Connected to a master server at 127.12.34.254:33097
I20260926 15:10:11.250665 22085 heartbeater.cc:464] Registering TS with master...
I20260926 15:10:11.251582 22085 heartbeater.cc:511] Master 127.12.34.254:33097 requested a full tablet report, sending...
I20260926 15:10:11.253043 21768 ts_manager.cc:194] Registered new tserver with Master: 6339ab94578644278e2d8dffa3505c0d (127.12.34.194:42887)
I20260926 15:10:11.253899 21768 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.34.194:60963
W20260926 15:10:11.563477 22088 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:10:11.563730 22088 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:10:11.563886 22088 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:10:11.570994 22088 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:10:11.571076 22088 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:10:11.571264 22088 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.34.195
I20260926 15:10:11.579960 22088 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.34.212:38875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790435318278681-12427-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790435318278681-12427-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.34.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790435318278681-12427-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.34.195
--webserver_port=0
--tserver_master_addrs=127.12.34.254:33097
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.34.195
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790435318278681-12427-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:10:11.581233 22088 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:10:11.582916 22088 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:10:11.595698 22096 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:10:11.595710 22093 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:10:11.595642 22094 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:10:11.598433 22088 server_base.cc:1034] running on GCE node
I20260926 15:10:11.598970 22088 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:10:11.599849 22088 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:10:11.601068 22088 hybrid_clock.cc:648] HybridClock initialized: now 1790435411601009 us; error 83 us; skew 500 ppm
I20260926 15:10:11.601424 22088 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 15:10:11.603511 22088 webserver.cc:460] Webserver started at http://127.12.34.195:40413/ using document root <none> and password file <none>
I20260926 15:10:11.604156 22088 fs_manager.cc:359] Metadata directory not provided
I20260926 15:10:11.604295 22088 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:10:11.604605 22088 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:10:11.607328 22088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790435318278681-12427-0/minicluster-data/ts-2/data/instance:
uuid: "0fe0d7b165044cde8dd1214dc39adb60"
format_stamp: "Formatted at 2026-09-26 15:10:11 on dist-test-slave-865v"
server_key: "863290ec9dcb613ae4592426699433b7"
server_key_iv: "598707245a81a4cfb1898a45cff2e681"
server_key_version: "encryptionkey@0"
I20260926 15:10:11.608093 22088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790435318278681-12427-0/minicluster-data/ts-2/wal/instance:
uuid: "0fe0d7b165044cde8dd1214dc39adb60"
format_stamp: "Formatted at 2026-09-26 15:10:11 on dist-test-slave-865v"
server_key: "863290ec9dcb613ae4592426699433b7"
server_key_iv: "598707245a81a4cfb1898a45cff2e681"
server_key_version: "encryptionkey@0"
I20260926 15:10:11.612815 22088 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260926 15:10:11.616127 22102 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:10:11.617507 22088 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 15:10:11.617692 22088 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790435318278681-12427-0/minicluster-data/ts-2/data,/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790435318278681-12427-0/minicluster-data/ts-2/wal
uuid: "0fe0d7b165044cde8dd1214dc39adb60"
format_stamp: "Formatted at 2026-09-26 15:10:11 on dist-test-slave-865v"
server_key: "863290ec9dcb613ae4592426699433b7"
server_key_iv: "598707245a81a4cfb1898a45cff2e681"
server_key_version: "encryptionkey@0"
I20260926 15:10:11.617911 22088 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790435318278681-12427-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790435318278681-12427-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790435318278681-12427-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:10:11.636780 22088 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:10:11.637621 22088 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:10:11.637910 22088 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:10:11.638839 22088 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:10:11.640221 22088 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:10:11.640343 22088 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:10:11.640476 22088 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:10:11.640563 22088 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:10:11.673207 22088 rpc_server.cc:307] RPC server started. Bound to: 127.12.34.195:45577
I20260926 15:10:11.673286 22214 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.34.195:45577 every 8 connection(s)
I20260926 15:10:11.674475 22088 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790435318278681-12427-0/minicluster-data/ts-2/data/info.pb
I20260926 15:10:11.675493 12427 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8uoJGv/build/asan/bin/kudu as pid 22088
I20260926 15:10:11.675642 12427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790435318278681-12427-0/minicluster-data/ts-2/wal/instance
I20260926 15:10:11.676213 12427 external_mini_cluster.cc:1428] Setting key ac18bac6b7e14b10ce730e0c43be199d
I20260926 15:10:11.687016 21956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:10:11.677479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.195:45393 (local address 127.12.34.254:33097)
0926 15:10:11.677680 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0926 15:10:11.677686 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:10:11.679367 (+ 1681us) server_negotiation.cc:407] Connection header received
0926 15:10:11.681020 (+ 1653us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:10:11.681024 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:10:11.681149 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:10:11.681275 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:10:11.682425 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:10:11.683135 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:10:11.684647 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:10:11.685034 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:10:11.685192 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:10:11.686414 (+ 1222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:10:11.686419 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:10:11.686422 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:10:11.686437 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:10:11.686463 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:10:11.686473 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:10:11.686540 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 15:10:11.686727 (+ 187us) server_negotiation.cc:299] Negotiation successful
0926 15:10:11.686794 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 15:10:11.687930 22217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:10:11.678112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:33097 (local address 127.12.34.195:45393)
0926 15:10:11.679163 (+ 1051us) negotiation.cc:107] Waiting for socket to connect
0926 15:10:11.679215 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 15:10:11.680773 (+ 1558us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:10:11.681533 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:10:11.681540 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:10:11.681670 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:10:11.682279 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:10:11.682295 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:10:11.683298 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:10:11.683302 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:10:11.684519 (+ 1217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:10:11.684531 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:10:11.685260 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:10:11.685265 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:10:11.685453 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:10:11.686138 (+ 685us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 15:10:11.686140 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:10:11.686174 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 15:10:11.686245 (+ 71us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:10:11.686557 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:10:11.686563 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:10:11.686569 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 15:10:11.686703 (+ 134us) client_negotiation.cc:240] Negotiation successful
0926 15:10:11.686969 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":303,"threads_started":1}
I20260926 15:10:11.689165 22215 heartbeater.cc:347] Connected to a master server at 127.12.34.254:33097
I20260926 15:10:11.689422 22215 heartbeater.cc:464] Registering TS with master...
I20260926 15:10:11.690194 22215 heartbeater.cc:511] Master 127.12.34.254:33097 requested a full tablet report, sending...
I20260926 15:10:11.691655 21768 ts_manager.cc:194] Registered new tserver with Master: 0fe0d7b165044cde8dd1214dc39adb60 (127.12.34.195:45577)
I20260926 15:10:11.692420 21768 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.34.195:45393
I20260926 15:10:11.703215 12427 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 15:10:11.709349 21956 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:10:11.707755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60808 (local address 127.12.34.254:33097)
0926 15:10:11.707928 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0926 15:10:11.707933 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:10:11.708444 (+ 511us) server_negotiation.cc:407] Connection header received
0926 15:10:11.708557 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:10:11.708561 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:10:11.708632 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:10:11.708713 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:10:11.709040 (+ 327us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60808: BlockingRecv error: recv got EOF from 127.0.0.1:60808 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 15:10:11.710147 12427 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 21827
I20260926 15:10:11.717685 22224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:10:11.710986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41620 (local address 127.12.34.193:37649)
0926 15:10:11.711496 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0926 15:10:11.711502 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:10:11.711525 (+ 23us) server_negotiation.cc:407] Connection header received
0926 15:10:11.711573 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:10:11.711579 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:10:11.711764 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:10:11.711880 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:10:11.712196 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:10:11.713285 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:10:11.714653 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:10:11.715256 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:10:11.715362 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:10:11.716297 (+ 935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:10:11.716304 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:10:11.716316 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:10:11.716877 (+ 561us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:10:11.716917 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:10:11.716934 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:10:11.717028 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 15:10:11.717222 (+ 194us) server_negotiation.cc:299] Negotiation successful
0926 15:10:11.717335 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":258,"threads_started":1}
I20260926 15:10:11.718807 21949 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:10:11.800992 12427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8uoJGv/build/asan/bin/kudu with pid 21827
I20260926 15:10:11.817976 12427 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 21958
I20260926 15:10:11.825379 22226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:10:11.818785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35750 (local address 127.12.34.194:42887)
0926 15:10:11.819360 (+ 575us) server_negotiation.cc:206] Beginning negotiation
0926 15:10:11.819368 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 15:10:11.819395 (+ 27us) server_negotiation.cc:407] Connection header received
0926 15:10:11.819455 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:10:11.819461 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:10:11.819685 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:10:11.819812 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:10:11.820132 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:10:11.821221 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:10:11.822519 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:10:11.823056 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:10:11.823153 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:10:11.824072 (+ 919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:10:11.824080 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:10:11.824091 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:10:11.824649 (+ 558us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:10:11.824692 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:10:11.824710 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:10:11.824795 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 15:10:11.824928 (+ 133us) server_negotiation.cc:299] Negotiation successful
0926 15:10:11.825026 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":306,"threads_started":1}
I20260926 15:10:11.826517 22080 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:10:11.907794 12427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8uoJGv/build/asan/bin/kudu with pid 21958
I20260926 15:10:11.924237 12427 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 22088
I20260926 15:10:11.932585 22228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:10:11.925241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39078 (local address 127.12.34.195:45577)
0926 15:10:11.925890 (+ 649us) server_negotiation.cc:206] Beginning negotiation
0926 15:10:11.925898 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 15:10:11.925926 (+ 28us) server_negotiation.cc:407] Connection header received
0926 15:10:11.925984 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:10:11.925991 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:10:11.926204 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:10:11.926468 (+ 264us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:10:11.926892 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:10:11.927979 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:10:11.929261 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:10:11.929799 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:10:11.929942 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:10:11.931087 (+ 1145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:10:11.931096 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:10:11.931109 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:10:11.931737 (+ 628us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:10:11.931779 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:10:11.931798 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:10:11.931924 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0926 15:10:11.932104 (+ 180us) server_negotiation.cc:299] Negotiation successful
0926 15:10:11.932250 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":464,"thread_start_us":329,"threads_started":1}
I20260926 15:10:11.933822 22210 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:10:12.019006 12427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8uoJGv/build/asan/bin/kudu with pid 22088
I20260926 15:10:12.035050 12427 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 21737
I20260926 15:10:12.036201 21798 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:10:12.136899 12427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8uoJGv/build/asan/bin/kudu with pid 21737
2026-09-26T15:10:12Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2438 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-26T15:10:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T15:10:12Z Disabled control of system clock
I20260926 15:10:12.207631 12427 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8uoJGv/build/asan/bin/kudu
/tmp/dist-test-task8uoJGv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790435318278681-12427-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790435318278681-12427-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790435318278681-12427-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.34.254:43863
--webserver_interface=127.12.34.254
--webserver_port=0
--builtin_ntp_servers=127.12.34.212:45509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.34.254:43863
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260926 15:10:12.535542 22234 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:10:12.535813 22234 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:10:12.535956 22234 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:10:12.543020 22234 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:10:12.543110 22234 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:10:12.543179 22234 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:10:12.543286 22234 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:10:12.543349 22234 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:10:12.552211 22234 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.34.212:45509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790435318278681-12427-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790435318278681-12427-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.34.254:43863
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.34.254:43863
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.34.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790435318278681-12427-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:10:12.553423 22234 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:10:12.555106 22234 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:10:12.566823 22239 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:10:12.566778 22240 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:10:12.566856 22242 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:10:12.567821 22234 server_base.cc:1034] running on GCE node
I20260926 15:10:12.568856 22234 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:10:12.570259 22234 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:10:12.571478 22234 hybrid_clock.cc:648] HybridClock initialized: now 1790435412571421 us; error 51 us; skew 500 ppm
I20260926 15:10:12.571902 22234 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 15:10:12.575143 22234 webserver.cc:460] Webserver started at http://127.12.34.254:34083/ using document root <none> and password file <none>
I20260926 15:10:12.575963 22234 fs_manager.cc:359] Metadata directory not provided
I20260926 15:10:12.576072 22234 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:10:12.576479 22234 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:10:12.579325 22234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790435318278681-12427-0/minicluster-data/master-0/data/instance:
uuid: "54dd1dd1b4b3473f9920b0acf5b9afa6"
format_stamp: "Formatted at 2026-09-26 15:10:12 on dist-test-slave-865v"
server_key: "590b7fb600222b05cc83bbe7f995c376"
server_key_iv: "0c62d914426abc49b1cd14f847fc781c"
server_key_version: "encryptionkey@0"
I20260926 15:10:12.580087 22234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790435318278681-12427-0/minicluster-data/master-0/wal/instance:
uuid: "54dd1dd1b4b3473f9920b0acf5b9afa6"
format_stamp: "Formatted at 2026-09-26 15:10:12 on dist-test-slave-865v"
server_key: "590b7fb600222b05cc83bbe7f995c376"
server_key_iv: "0c62d914426abc49b1cd14f847fc781c"
server_key_version: "encryptionkey@0"
I20260926 15:10:12.584820 22234 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260926 15:10:12.588050 22248 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:10:12.589313 22234 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 15:10:12.589480 22234 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790435318278681-12427-0/minicluster-data/master-0/data,/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790435318278681-12427-0/minicluster-data/master-0/wal
uuid: "54dd1dd1b4b3473f9920b0acf5b9afa6"
format_stamp: "Formatted at 2026-09-26 15:10:12 on dist-test-slave-865v"
server_key: "590b7fb600222b05cc83bbe7f995c376"
server_key_iv: "0c62d914426abc49b1cd14f847fc781c"
server_key_version: "encryptionkey@0"
I20260926 15:10:12.589668 22234 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790435318278681-12427-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790435318278681-12427-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790435318278681-12427-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:10:12.596586 22234 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:10:12.597440 22234 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:10:12.597769 22234 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:10:12.618127 22234 rpc_server.cc:307] RPC server started. Bound to: 127.12.34.254:43863
I20260926 15:10:12.618305 22299 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.34.254:43863 every 8 connection(s)
I20260926 15:10:12.619416 22234 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
I20260926 15:10:12.623086 22300 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:10:12.628661 12427 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8uoJGv/build/asan/bin/kudu as pid 22234
I20260926 15:10:12.628880 12427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790435318278681-12427-0/minicluster-data/master-0/wal/instance
I20260926 15:10:12.629666 12427 external_mini_cluster.cc:1428] Setting key 7321559c2a08012fe6a991cdd3bfe95c
I20260926 15:10:12.633848 22300 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54dd1dd1b4b3473f9920b0acf5b9afa6: Bootstrap starting.
I20260926 15:10:12.637276 22300 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 54dd1dd1b4b3473f9920b0acf5b9afa6: Neither blocks nor log segments found. Creating new log.
I20260926 15:10:12.637951 22303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:10:12.631749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33982 (local address 127.12.34.254:43863)
0926 15:10:12.632744 (+ 995us) server_negotiation.cc:206] Beginning negotiation
0926 15:10:12.632754 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 15:10:12.632794 (+ 40us) server_negotiation.cc:407] Connection header received
0926 15:10:12.634446 (+ 1652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:10:12.634466 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:10:12.634833 (+ 367us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:10:12.635141 (+ 308us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:10:12.636218 (+ 1077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:10:12.636224 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:10:12.636260 (+ 36us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:10:12.636278 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:10:12.636358 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:10:12.636375 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:10:12.636518 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0926 15:10:12.636683 (+ 165us) server_negotiation.cc:299] Negotiation successful
0926 15:10:12.636963 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":609,"thread_start_us":414,"threads_started":1}
I20260926 15:10:12.638335 22300 log.cc:824] T 00000000000000000000000000000000 P 54dd1dd1b4b3473f9920b0acf5b9afa6: Log is configured to *not* fsync() on all Append() calls
I20260926 15:10:12.641324 22300 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54dd1dd1b4b3473f9920b0acf5b9afa6: No bootstrap required, opened a new log
I20260926 15:10:12.648061 22300 raft_consensus.cc:348] T 00000000000000000000000000000000 P 54dd1dd1b4b3473f9920b0acf5b9afa6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54dd1dd1b4b3473f9920b0acf5b9afa6" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 43863 } }
I20260926 15:10:12.648375 22300 raft_consensus.cc:374] T 00000000000000000000000000000000 P 54dd1dd1b4b3473f9920b0acf5b9afa6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:10:12.648455 22300 raft_consensus.cc:729] T 00000000000000000000000000000000 P 54dd1dd1b4b3473f9920b0acf5b9afa6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54dd1dd1b4b3473f9920b0acf5b9afa6, State: Initialized, Role: FOLLOWER
I20260926 15:10:12.649061 22300 consensus_queue.cc:260] T 00000000000000000000000000000000 P 54dd1dd1b4b3473f9920b0acf5b9afa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54dd1dd1b4b3473f9920b0acf5b9afa6" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 43863 } }
I20260926 15:10:12.649333 22300 raft_consensus.cc:388] T 00000000000000000000000000000000 P 54dd1dd1b4b3473f9920b0acf5b9afa6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 15:10:12.649461 22300 raft_consensus.cc:482] T 00000000000000000000000000000000 P 54dd1dd1b4b3473f9920b0acf5b9afa6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 15:10:12.649616 22300 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 54dd1dd1b4b3473f9920b0acf5b9afa6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:10:12.651711 22300 raft_consensus.cc:504] T 00000000000000000000000000000000 P 54dd1dd1b4b3473f9920b0acf5b9afa6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54dd1dd1b4b3473f9920b0acf5b9afa6" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 43863 } }
I20260926 15:10:12.652184 22300 leader_election.cc:302] T 00000000000000000000000000000000 P 54dd1dd1b4b3473f9920b0acf5b9afa6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 54dd1dd1b4b3473f9920b0acf5b9afa6; no voters:
I20260926 15:10:12.652741 22300 leader_election.cc:288] T 00000000000000000000000000000000 P 54dd1dd1b4b3473f9920b0acf5b9afa6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 15:10:12.652896 22305 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 54dd1dd1b4b3473f9920b0acf5b9afa6 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:10:12.653637 22305 raft_consensus.cc:686] T 00000000000000000000000000000000 P 54dd1dd1b4b3473f9920b0acf5b9afa6 [term 1 LEADER]: Becoming Leader. State: Replica: 54dd1dd1b4b3473f9920b0acf5b9afa6, State: Running, Role: LEADER
I20260926 15:10:12.654156 22305 consensus_queue.cc:237] T 00000000000000000000000000000000 P 54dd1dd1b4b3473f9920b0acf5b9afa6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54dd1dd1b4b3473f9920b0acf5b9afa6" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 43863 } }
I20260926 15:10:12.654836 22300 sys_catalog.cc:564] T 00000000000000000000000000000000 P 54dd1dd1b4b3473f9920b0acf5b9afa6 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:10:12.656670 22307 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54dd1dd1b4b3473f9920b0acf5b9afa6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 54dd1dd1b4b3473f9920b0acf5b9afa6. Latest consensus state: current_term: 1 leader_uuid: "54dd1dd1b4b3473f9920b0acf5b9afa6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54dd1dd1b4b3473f9920b0acf5b9afa6" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 43863 } } }
I20260926 15:10:12.657004 22307 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54dd1dd1b4b3473f9920b0acf5b9afa6 [sys.catalog]: This master's current role is: LEADER
I20260926 15:10:12.658102 22306 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54dd1dd1b4b3473f9920b0acf5b9afa6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "54dd1dd1b4b3473f9920b0acf5b9afa6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54dd1dd1b4b3473f9920b0acf5b9afa6" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 43863 } } }
I20260926 15:10:12.658496 22306 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54dd1dd1b4b3473f9920b0acf5b9afa6 [sys.catalog]: This master's current role is: LEADER
I20260926 15:10:12.665418 22311 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:10:12.670607 22311 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 15:10:12.679368 22311 catalog_manager.cc:1347] Generated new cluster ID: 83dac423e77642bea40e69e0f892efe1
I20260926 15:10:12.679508 22311 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 15:10:12.705827 22311 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 15:10:12.706351 22311 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:10:12.733148 22311 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 54dd1dd1b4b3473f9920b0acf5b9afa6: Generated new TSK 0
I20260926 15:10:12.734239 22311 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 15:10:12.753368 12427 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8uoJGv/build/asan/bin/kudu
/tmp/dist-test-task8uoJGv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790435318278681-12427-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790435318278681-12427-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790435318278681-12427-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790435318278681-12427-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.34.193:0
--local_ip_for_outbound_sockets=127.12.34.193
--webserver_interface=127.12.34.193
--webserver_port=0
--tserver_master_addrs=127.12.34.254:43863
--builtin_ntp_servers=127.12.34.212:45509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260926 15:10:13.083313 22324 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:10:13.083628 22324 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:10:13.083792 22324 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:10:13.091352 22324 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:10:13.091496 22324 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:10:13.091686 22324 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.34.193
I20260926 15:10:13.103991 22324 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.34.212:45509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790435318278681-12427-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790435318278681-12427-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.34.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790435318278681-12427-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.34.193
--webserver_port=0
--tserver_master_addrs=127.12.34.254:43863
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.34.193
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790435318278681-12427-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:10:13.105727 22324 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:10:13.107860 22324 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:10:13.122758 22329 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:10:13.122780 22330 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:10:13.123030 22332 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:10:13.123498 22324 server_base.cc:1034] running on GCE node
I20260926 15:10:13.124094 22324 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:10:13.125000 22324 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:10:13.126241 22324 hybrid_clock.cc:648] HybridClock initialized: now 1790435413126192 us; error 34 us; skew 500 ppm
I20260926 15:10:13.126626 22324 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 15:10:13.128960 22324 webserver.cc:460] Webserver started at http://127.12.34.193:35241/ using document root <none> and password file <none>
I20260926 15:10:13.129652 22324 fs_manager.cc:359] Metadata directory not provided
I20260926 15:10:13.129741 22324 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:10:13.129981 22324 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:10:13.134505 22324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790435318278681-12427-0/minicluster-data/ts-0/data/instance:
uuid: "472525dde9cf4a7cad35db8845a9ff4f"
format_stamp: "Formatted at 2026-09-26 15:10:13 on dist-test-slave-865v"
server_key: "82f6bba25b8ee1dbd5a3bfa4919f5e67"
server_key_iv: "860e7939ac4ae7871e43f52ff3181c01"
server_key_version: "encryptionkey@0"
I20260926 15:10:13.135219 22324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790435318278681-12427-0/minicluster-data/ts-0/wal/instance:
uuid: "472525dde9cf4a7cad35db8845a9ff4f"
format_stamp: "Formatted at 2026-09-26 15:10:13 on dist-test-slave-865v"
server_key: "82f6bba25b8ee1dbd5a3bfa4919f5e67"
server_key_iv: "860e7939ac4ae7871e43f52ff3181c01"
server_key_version: "encryptionkey@0"
I20260926 15:10:13.140307 22324 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 15:10:13.143589 22338 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:10:13.144896 22324 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 15:10:13.145133 22324 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790435318278681-12427-0/minicluster-data/ts-0/data,/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790435318278681-12427-0/minicluster-data/ts-0/wal
uuid: "472525dde9cf4a7cad35db8845a9ff4f"
format_stamp: "Formatted at 2026-09-26 15:10:13 on dist-test-slave-865v"
server_key: "82f6bba25b8ee1dbd5a3bfa4919f5e67"
server_key_iv: "860e7939ac4ae7871e43f52ff3181c01"
server_key_version: "encryptionkey@0"
I20260926 15:10:13.145354 22324 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790435318278681-12427-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790435318278681-12427-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790435318278681-12427-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:10:13.150833 22324 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:10:13.151563 22324 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:10:13.151821 22324 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:10:13.152751 22324 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:10:13.154130 22324 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:10:13.154215 22324 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:10:13.154345 22324 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:10:13.154446 22324 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:10:13.187213 22324 rpc_server.cc:307] RPC server started. Bound to: 127.12.34.193:43723
I20260926 15:10:13.187309 22450 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.34.193:43723 every 8 connection(s)
I20260926 15:10:13.188495 22324 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790435318278681-12427-0/minicluster-data/ts-0/data/info.pb
I20260926 15:10:13.196817 22453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:10:13.191186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.193:41987 (local address 127.12.34.254:43863)
0926 15:10:13.191673 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0926 15:10:13.191677 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:10:13.192692 (+ 1015us) server_negotiation.cc:407] Connection header received
0926 15:10:13.194499 (+ 1807us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:10:13.194505 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:10:13.194580 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:10:13.194717 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:10:13.196024 (+ 1307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:10:13.196030 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:10:13.196035 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:10:13.196053 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:10:13.196100 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:10:13.196111 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:10:13.196234 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0926 15:10:13.196430 (+ 196us) server_negotiation.cc:299] Negotiation successful
0926 15:10:13.196496 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":257,"threads_started":1}
I20260926 15:10:13.197706 22454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:10:13.191640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:43863 (local address 127.12.34.193:41987)
0926 15:10:13.192523 (+ 883us) negotiation.cc:107] Waiting for socket to connect
0926 15:10:13.192568 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 15:10:13.194247 (+ 1679us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:10:13.194960 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:10:13.194968 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:10:13.195053 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:10:13.195749 (+ 696us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 15:10:13.195752 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:10:13.195786 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 15:10:13.195863 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:10:13.196204 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:10:13.196214 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:10:13.196221 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 15:10:13.196496 (+ 275us) client_negotiation.cc:240] Negotiation successful
0926 15:10:13.196726 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":260,"threads_started":1}
I20260926 15:10:13.198102 12427 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8uoJGv/build/asan/bin/kudu as pid 22324
I20260926 15:10:13.198381 12427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790435318278681-12427-0/minicluster-data/ts-0/wal/instance
I20260926 15:10:13.199199 12427 external_mini_cluster.cc:1428] Setting key a8dc918871a4cbf1ff89958ebbb5744d
I20260926 15:10:13.199486 22451 heartbeater.cc:347] Connected to a master server at 127.12.34.254:43863
I20260926 15:10:13.199769 22451 heartbeater.cc:464] Registering TS with master...
I20260926 15:10:13.200388 22451 heartbeater.cc:511] Master 127.12.34.254:43863 requested a full tablet report, sending...
I20260926 15:10:13.202111 12427 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8uoJGv/build/asan/bin/kudu
/tmp/dist-test-task8uoJGv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790435318278681-12427-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790435318278681-12427-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790435318278681-12427-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790435318278681-12427-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.34.194:0
--local_ip_for_outbound_sockets=127.12.34.194
--webserver_interface=127.12.34.194
--webserver_port=0
--tserver_master_addrs=127.12.34.254:43863
--builtin_ntp_servers=127.12.34.212:45509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260926 15:10:13.202536 22265 ts_manager.cc:194] Registered new tserver with Master: 472525dde9cf4a7cad35db8845a9ff4f (127.12.34.193:43723)
W20260926 15:10:13.532691 22455 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:10:13.533017 22455 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:10:13.533252 22455 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:10:13.540382 22455 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:10:13.540532 22455 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:10:13.540712 22455 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.34.194
I20260926 15:10:13.549697 22455 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.34.212:45509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790435318278681-12427-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790435318278681-12427-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.34.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790435318278681-12427-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.34.194
--webserver_port=0
--tserver_master_addrs=127.12.34.254:43863
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.34.194
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790435318278681-12427-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:10:13.550956 22455 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:10:13.552635 22455 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:10:13.567050 22460 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:10:13.567241 22461 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:10:13.567567 22463 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:10:13.569216 22455 server_base.cc:1034] running on GCE node
I20260926 15:10:13.570257 22455 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:10:13.571630 22455 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:10:13.572901 22455 hybrid_clock.cc:648] HybridClock initialized: now 1790435413572840 us; error 56 us; skew 500 ppm
I20260926 15:10:13.573455 22455 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 15:10:13.576136 22455 webserver.cc:460] Webserver started at http://127.12.34.194:35863/ using document root <none> and password file <none>
I20260926 15:10:13.576913 22455 fs_manager.cc:359] Metadata directory not provided
I20260926 15:10:13.576999 22455 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:10:13.577343 22455 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:10:13.580118 22455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790435318278681-12427-0/minicluster-data/ts-1/data/instance:
uuid: "870e5ca230c941128516549c3f68c4a0"
format_stamp: "Formatted at 2026-09-26 15:10:13 on dist-test-slave-865v"
server_key: "45b6a6d940db6b6da4574f4698912a64"
server_key_iv: "d6bdfea6ef42c8fc126c1636ec2527e9"
server_key_version: "encryptionkey@0"
I20260926 15:10:13.580889 22455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790435318278681-12427-0/minicluster-data/ts-1/wal/instance:
uuid: "870e5ca230c941128516549c3f68c4a0"
format_stamp: "Formatted at 2026-09-26 15:10:13 on dist-test-slave-865v"
server_key: "45b6a6d940db6b6da4574f4698912a64"
server_key_iv: "d6bdfea6ef42c8fc126c1636ec2527e9"
server_key_version: "encryptionkey@0"
I20260926 15:10:13.586437 22455 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260926 15:10:13.590272 22469 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:10:13.591742 22455 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 15:10:13.591872 22455 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790435318278681-12427-0/minicluster-data/ts-1/data,/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790435318278681-12427-0/minicluster-data/ts-1/wal
uuid: "870e5ca230c941128516549c3f68c4a0"
format_stamp: "Formatted at 2026-09-26 15:10:13 on dist-test-slave-865v"
server_key: "45b6a6d940db6b6da4574f4698912a64"
server_key_iv: "d6bdfea6ef42c8fc126c1636ec2527e9"
server_key_version: "encryptionkey@0"
I20260926 15:10:13.592052 22455 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790435318278681-12427-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790435318278681-12427-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790435318278681-12427-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:10:13.597959 22455 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:10:13.598717 22455 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:10:13.598953 22455 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:10:13.599951 22455 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:10:13.601879 22455 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:10:13.601969 22455 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:10:13.602041 22455 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:10:13.602094 22455 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:10:13.643354 22455 rpc_server.cc:307] RPC server started. Bound to: 127.12.34.194:36601
I20260926 15:10:13.643400 22581 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.34.194:36601 every 8 connection(s)
I20260926 15:10:13.644855 22455 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790435318278681-12427-0/minicluster-data/ts-1/data/info.pb
I20260926 15:10:13.653527 22453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:10:13.647651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.194:59495 (local address 127.12.34.254:43863)
0926 15:10:13.647822 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0926 15:10:13.647827 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:10:13.649345 (+ 1518us) server_negotiation.cc:407] Connection header received
0926 15:10:13.651025 (+ 1680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:10:13.651030 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:10:13.651081 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:10:13.651169 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:10:13.652790 (+ 1621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:10:13.652794 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:10:13.652798 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:10:13.652813 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:10:13.652845 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:10:13.652856 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:10:13.652965 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 15:10:13.653284 (+ 319us) server_negotiation.cc:299] Negotiation successful
0926 15:10:13.653337 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 15:10:13.654467 22584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:10:13.648195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:43863 (local address 127.12.34.194:59495)
0926 15:10:13.649205 (+ 1010us) negotiation.cc:107] Waiting for socket to connect
0926 15:10:13.649239 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 15:10:13.650771 (+ 1532us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:10:13.651383 (+ 612us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:10:13.651391 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:10:13.651471 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:10:13.652263 (+ 792us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 15:10:13.652266 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:10:13.652325 (+ 59us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 15:10:13.652459 (+ 134us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:10:13.652954 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:10:13.652962 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:10:13.652967 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 15:10:13.653325 (+ 358us) client_negotiation.cc:240] Negotiation successful
0926 15:10:13.653548 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":278,"threads_started":1}
I20260926 15:10:13.655143 12427 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8uoJGv/build/asan/bin/kudu as pid 22455
I20260926 15:10:13.655321 12427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790435318278681-12427-0/minicluster-data/ts-1/wal/instance
I20260926 15:10:13.655769 12427 external_mini_cluster.cc:1428] Setting key 6f9c8cf36af141478e7d656cb2bb004e
I20260926 15:10:13.656066 22582 heartbeater.cc:347] Connected to a master server at 127.12.34.254:43863
I20260926 15:10:13.656400 22582 heartbeater.cc:464] Registering TS with master...
I20260926 15:10:13.657071 22582 heartbeater.cc:511] Master 127.12.34.254:43863 requested a full tablet report, sending...
I20260926 15:10:13.658550 22265 ts_manager.cc:194] Registered new tserver with Master: 870e5ca230c941128516549c3f68c4a0 (127.12.34.194:36601)
I20260926 15:10:13.658644 12427 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8uoJGv/build/asan/bin/kudu
/tmp/dist-test-task8uoJGv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790435318278681-12427-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790435318278681-12427-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790435318278681-12427-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790435318278681-12427-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.34.195:0
--local_ip_for_outbound_sockets=127.12.34.195
--webserver_interface=127.12.34.195
--webserver_port=0
--tserver_master_addrs=127.12.34.254:43863
--builtin_ntp_servers=127.12.34.212:45509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260926 15:10:13.973752 22585 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:10:13.974051 22585 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:10:13.974216 22585 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:10:13.981763 22585 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:10:13.981896 22585 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:10:13.982079 22585 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.34.195
I20260926 15:10:13.991098 22585 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.34.212:45509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790435318278681-12427-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790435318278681-12427-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.34.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790435318278681-12427-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.34.195
--webserver_port=0
--tserver_master_addrs=127.12.34.254:43863
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.34.195
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790435318278681-12427-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:10:13.992369 22585 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:10:13.994330 22585 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:10:14.007747 22590 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:10:14.007772 22591 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:10:14.007900 22593 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:10:14.008596 22585 server_base.cc:1034] running on GCE node
I20260926 15:10:14.009265 22585 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:10:14.010200 22585 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:10:14.011435 22585 hybrid_clock.cc:648] HybridClock initialized: now 1790435414011406 us; error 43 us; skew 500 ppm
I20260926 15:10:14.011765 22585 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 15:10:14.014138 22585 webserver.cc:460] Webserver started at http://127.12.34.195:37949/ using document root <none> and password file <none>
I20260926 15:10:14.014803 22585 fs_manager.cc:359] Metadata directory not provided
I20260926 15:10:14.014930 22585 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:10:14.015267 22585 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:10:14.018134 22585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790435318278681-12427-0/minicluster-data/ts-2/data/instance:
uuid: "c0a1b3cc593d475fbdefe8e0d579bfed"
format_stamp: "Formatted at 2026-09-26 15:10:14 on dist-test-slave-865v"
server_key: "4e3e36345e8088ed5db5a97edcf7f56a"
server_key_iv: "f57c7202254a5a155d0156b646df54b4"
server_key_version: "encryptionkey@0"
I20260926 15:10:14.018898 22585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790435318278681-12427-0/minicluster-data/ts-2/wal/instance:
uuid: "c0a1b3cc593d475fbdefe8e0d579bfed"
format_stamp: "Formatted at 2026-09-26 15:10:14 on dist-test-slave-865v"
server_key: "4e3e36345e8088ed5db5a97edcf7f56a"
server_key_iv: "f57c7202254a5a155d0156b646df54b4"
server_key_version: "encryptionkey@0"
I20260926 15:10:14.023764 22585 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 15:10:14.027019 22599 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:10:14.028327 22585 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 15:10:14.028493 22585 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790435318278681-12427-0/minicluster-data/ts-2/data,/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790435318278681-12427-0/minicluster-data/ts-2/wal
uuid: "c0a1b3cc593d475fbdefe8e0d579bfed"
format_stamp: "Formatted at 2026-09-26 15:10:14 on dist-test-slave-865v"
server_key: "4e3e36345e8088ed5db5a97edcf7f56a"
server_key_iv: "f57c7202254a5a155d0156b646df54b4"
server_key_version: "encryptionkey@0"
I20260926 15:10:14.028716 22585 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790435318278681-12427-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790435318278681-12427-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790435318278681-12427-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:10:14.034016 22585 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:10:14.034744 22585 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:10:14.034996 22585 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:10:14.035820 22585 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:10:14.037120 22585 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:10:14.037245 22585 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:10:14.037369 22585 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:10:14.037460 22585 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:10:14.068627 22585 rpc_server.cc:307] RPC server started. Bound to: 127.12.34.195:45415
I20260926 15:10:14.068742 22711 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.34.195:45415 every 8 connection(s)
I20260926 15:10:14.070137 22585 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790435318278681-12427-0/minicluster-data/ts-2/data/info.pb
I20260926 15:10:14.078349 22453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:10:14.072995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.195:59447 (local address 127.12.34.254:43863)
0926 15:10:14.073220 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0926 15:10:14.073224 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:10:14.074552 (+ 1328us) server_negotiation.cc:407] Connection header received
0926 15:10:14.076230 (+ 1678us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:10:14.076235 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:10:14.076309 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:10:14.076416 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:10:14.077786 (+ 1370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:10:14.077790 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:10:14.077793 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:10:14.077808 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:10:14.077837 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:10:14.077848 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:10:14.077947 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 15:10:14.078102 (+ 155us) server_negotiation.cc:299] Negotiation successful
0926 15:10:14.078149 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260926 15:10:14.079211 22714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:10:14.073524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:43863 (local address 127.12.34.195:59447)
0926 15:10:14.074385 (+ 861us) negotiation.cc:107] Waiting for socket to connect
0926 15:10:14.074428 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 15:10:14.075981 (+ 1553us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:10:14.076622 (+ 641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:10:14.076630 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:10:14.076715 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:10:14.077482 (+ 767us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 15:10:14.077486 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:10:14.077530 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 15:10:14.077626 (+ 96us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:10:14.077934 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:10:14.077941 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:10:14.077946 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 15:10:14.078146 (+ 200us) client_negotiation.cc:240] Negotiation successful
0926 15:10:14.078317 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":266,"threads_started":1}
I20260926 15:10:14.080171 12427 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8uoJGv/build/asan/bin/kudu as pid 22585
I20260926 15:10:14.080353 12427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790435318278681-12427-0/minicluster-data/ts-2/wal/instance
I20260926 15:10:14.080813 12427 external_mini_cluster.cc:1428] Setting key 64141c1e74aaa2c7779f8354f6dddf40
I20260926 15:10:14.080902 22712 heartbeater.cc:347] Connected to a master server at 127.12.34.254:43863
I20260926 15:10:14.081243 22712 heartbeater.cc:464] Registering TS with master...
I20260926 15:10:14.081874 22712 heartbeater.cc:511] Master 127.12.34.254:43863 requested a full tablet report, sending...
I20260926 15:10:14.083477 22265 ts_manager.cc:194] Registered new tserver with Master: c0a1b3cc593d475fbdefe8e0d579bfed (127.12.34.195:45415)
I20260926 15:10:14.084578 12427 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 15:10:14.090432 22453 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:10:14.088962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33986 (local address 127.12.34.254:43863)
0926 15:10:14.089169 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0926 15:10:14.089174 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:10:14.089630 (+ 456us) server_negotiation.cc:407] Connection header received
0926 15:10:14.089737 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:10:14.089741 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:10:14.089787 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:10:14.089893 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:10:14.090226 (+ 333us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33986: BlockingRecv error: recv got EOF from 127.0.0.1:33986 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 15:10:14.090792 12427 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 22324
I20260926 15:10:14.094233 22721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:10:14.091812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58544 (local address 127.12.34.193:43723)
0926 15:10:14.092307 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0926 15:10:14.092313 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:10:14.092336 (+ 23us) server_negotiation.cc:407] Connection header received
0926 15:10:14.092387 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:10:14.092396 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:10:14.092570 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:10:14.092694 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:10:14.093564 (+ 870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:10:14.093570 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:10:14.093580 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:10:14.093597 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:10:14.093641 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:10:14.093656 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:10:14.093753 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 15:10:14.093936 (+ 183us) server_negotiation.cc:299] Negotiation successful
0926 15:10:14.094011 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":255,"threads_started":1}
I20260926 15:10:14.095386 22446 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:10:14.175314 12427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8uoJGv/build/asan/bin/kudu with pid 22324
I20260926 15:10:14.192941 12427 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 22455
I20260926 15:10:14.196688 22723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:10:14.194040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50290 (local address 127.12.34.194:36601)
0926 15:10:14.194558 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0926 15:10:14.194564 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:10:14.194590 (+ 26us) server_negotiation.cc:407] Connection header received
0926 15:10:14.194655 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:10:14.194661 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:10:14.194870 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:10:14.195061 (+ 191us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:10:14.196025 (+ 964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:10:14.196032 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:10:14.196044 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:10:14.196057 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:10:14.196100 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:10:14.196115 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:10:14.196221 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 15:10:14.196373 (+ 152us) server_negotiation.cc:299] Negotiation successful
0926 15:10:14.196448 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":246,"threads_started":1}
I20260926 15:10:14.197896 22577 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:10:14.278292 12427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8uoJGv/build/asan/bin/kudu with pid 22455
I20260926 15:10:14.295609 12427 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 22585
I20260926 15:10:14.299960 22725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:10:14.296822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36614 (local address 127.12.34.195:45415)
0926 15:10:14.297532 (+ 710us) server_negotiation.cc:206] Beginning negotiation
0926 15:10:14.297539 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:10:14.297571 (+ 32us) server_negotiation.cc:407] Connection header received
0926 15:10:14.297631 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:10:14.297637 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:10:14.297868 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:10:14.298042 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:10:14.299163 (+ 1121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:10:14.299175 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:10:14.299185 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:10:14.299210 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:10:14.299272 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:10:14.299289 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:10:14.299480 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0926 15:10:14.299553 (+ 73us) server_negotiation.cc:299] Negotiation successful
0926 15:10:14.299650 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":561,"thread_start_us":395,"threads_started":1}
I20260926 15:10:14.301293 22707 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:10:14.385025 12427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8uoJGv/build/asan/bin/kudu with pid 22585
I20260926 15:10:14.401021 12427 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 22234
I20260926 15:10:14.402313 22295 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:10:14.503758 12427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8uoJGv/build/asan/bin/kudu with pid 22234
2026-09-26T15:10:14Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2367 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435318278681-12427-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 15:10:14 dist-test-slave-865v krb5kdc[22730](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 15:10:14 dist-test-slave-865v krb5kdc[22730](info): set up 2 sockets
Sep 26 15:10:14 dist-test-slave-865v krb5kdc[22730](info): commencing operation
krb5kdc: starting...
W20260926 15:10:15.633837 12427 mini_kdc.cc:121] Time spent starting KDC: real 1.089s user 0.000s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435318278681-12427-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435318278681-12427-0/krb5kdc/test-user.keytab.
Sep 26 15:10:15 dist-test-slave-865v krb5kdc[22730](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435415, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T15:10:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T15:10:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.34.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.34.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.34.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.254.keytab.
Entry for principal kudu/127.12.34.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.254.keytab.
I20260926 15:10:15.856278 12427 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8uoJGv/build/asan/bin/kudu
/tmp/dist-test-task8uoJGv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435318278681-12427-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435318278681-12427-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435318278681-12427-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.34.254:43971
--webserver_interface=127.12.34.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.254.keytab
--principal=kudu/127.12.34.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.34.212:44131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.34.254:43971
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435318278681-12427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435318278681-12427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435318278681-12427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:10:16.169113 22743 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:10:16.169549 22743 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:10:16.169713 22743 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:10:16.177134 22743 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:10:16.177289 22743 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:10:16.177400 22743 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:10:16.177510 22743 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:10:16.177598 22743 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:10:16.186990 22743 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.34.212:44131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435318278681-12427-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435318278681-12427-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.34.254:43971
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.254.keytab
--principal=kudu/127.12.34.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.34.254:43971
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.34.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435318278681-12427-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:10:16.188328 22743 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:10:16.189606 22743 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:10:16.200953 22748 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:10:16.200978 22749 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:10:16.201115 22751 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:10:16.202142 22743 server_base.cc:1034] running on GCE node
I20260926 15:10:16.202877 22743 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:10:16.204200 22743 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:10:16.205425 22743 hybrid_clock.cc:648] HybridClock initialized: now 1790435416205405 us; error 45 us; skew 500 ppm
I20260926 15:10:16.205777 22743 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:10:16 dist-test-slave-865v krb5kdc[22730](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435416, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:10:16.216838 22743 init.cc:377] Logged in from keytab as kudu/127.12.34.254@KRBTEST.COM (short username kudu)
I20260926 15:10:16.218490 22743 webserver.cc:460] Webserver started at http://127.12.34.254:46425/ using document root <none> and password file <none>
I20260926 15:10:16.219177 22743 fs_manager.cc:359] Metadata directory not provided
I20260926 15:10:16.219331 22743 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:10:16.219638 22743 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:10:16.222595 22743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435318278681-12427-0/minicluster-data/master-0/data/instance:
uuid: "9d2d5749ac6a41b68188b1f8ffeef057"
format_stamp: "Formatted at 2026-09-26 15:10:16 on dist-test-slave-865v"
server_key: "1d79655d76d9156d5eb21c6828536348"
server_key_iv: "bb4aeec5ff8e56013c2d10008d7a81f7"
server_key_version: "encryptionkey@0"
I20260926 15:10:16.223335 22743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435318278681-12427-0/minicluster-data/master-0/wal/instance:
uuid: "9d2d5749ac6a41b68188b1f8ffeef057"
format_stamp: "Formatted at 2026-09-26 15:10:16 on dist-test-slave-865v"
server_key: "1d79655d76d9156d5eb21c6828536348"
server_key_iv: "bb4aeec5ff8e56013c2d10008d7a81f7"
server_key_version: "encryptionkey@0"
I20260926 15:10:16.228219 22743 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260926 15:10:16.231503 22758 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:10:16.232741 22743 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260926 15:10:16.232921 22743 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435318278681-12427-0/minicluster-data/master-0/data,/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435318278681-12427-0/minicluster-data/master-0/wal
uuid: "9d2d5749ac6a41b68188b1f8ffeef057"
format_stamp: "Formatted at 2026-09-26 15:10:16 on dist-test-slave-865v"
server_key: "1d79655d76d9156d5eb21c6828536348"
server_key_iv: "bb4aeec5ff8e56013c2d10008d7a81f7"
server_key_version: "encryptionkey@0"
I20260926 15:10:16.233134 22743 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435318278681-12427-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435318278681-12427-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435318278681-12427-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:10:16.253000 22743 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:10:16.256866 22743 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:10:16.257164 22743 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:10:16.275206 22809 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.34.254:43971 every 8 connection(s)
I20260926 15:10:16.275216 22743 rpc_server.cc:307] RPC server started. Bound to: 127.12.34.254:43971
I20260926 15:10:16.276608 22743 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
I20260926 15:10:16.276974 12427 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8uoJGv/build/asan/bin/kudu as pid 22743
I20260926 15:10:16.277271 12427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435318278681-12427-0/minicluster-data/master-0/wal/instance
I20260926 15:10:16.278025 12427 external_mini_cluster.cc:1428] Setting key 37534f775cf33f477498364202794962
I20260926 15:10:16.280865 22810 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 26 15:10:16 dist-test-slave-865v krb5kdc[22730](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435415, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.34.254@KRBTEST.COM
I20260926 15:10:16.293812 22810 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d2d5749ac6a41b68188b1f8ffeef057: Bootstrap starting.
I20260926 15:10:16.297344 22810 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9d2d5749ac6a41b68188b1f8ffeef057: Neither blocks nor log segments found. Creating new log.
I20260926 15:10:16.298345 22810 log.cc:824] T 00000000000000000000000000000000 P 9d2d5749ac6a41b68188b1f8ffeef057: Log is configured to *not* fsync() on all Append() calls
I20260926 15:10:16.301216 22810 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d2d5749ac6a41b68188b1f8ffeef057: No bootstrap required, opened a new log
I20260926 15:10:16.301891 22813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:10:16.280029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57046 (local address 127.12.34.254:43971)
0926 15:10:16.280755 (+ 726us) server_negotiation.cc:206] Beginning negotiation
0926 15:10:16.280770 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 15:10:16.280820 (+ 50us) server_negotiation.cc:407] Connection header received
0926 15:10:16.283072 (+ 2252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:10:16.283098 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:10:16.283546 (+ 448us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:10:16.283995 (+ 449us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:10:16.285412 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:10:16.287435 (+ 2023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:10:16.288719 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:10:16.289813 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:10:16.289999 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:10:16.294715 (+ 4716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:10:16.294729 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:10:16.294751 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:10:16.294813 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:10:16.297939 (+ 3126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:10:16.298632 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:10:16.298640 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:10:16.298645 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:10:16.298807 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:10:16.299193 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:10:16.299199 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:10:16.299202 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:10:16.299720 (+ 518us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:10:16.300020 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0926 15:10:16.300443 (+ 423us) server_negotiation.cc:299] Negotiation successful
0926 15:10:16.300827 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":521,"thread_start_us":303,"threads_started":1}
I20260926 15:10:16.309713 22810 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9d2d5749ac6a41b68188b1f8ffeef057 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d2d5749ac6a41b68188b1f8ffeef057" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 43971 } }
I20260926 15:10:16.310176 22810 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9d2d5749ac6a41b68188b1f8ffeef057 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:10:16.310305 22810 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9d2d5749ac6a41b68188b1f8ffeef057 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d2d5749ac6a41b68188b1f8ffeef057, State: Initialized, Role: FOLLOWER
I20260926 15:10:16.310859 22810 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9d2d5749ac6a41b68188b1f8ffeef057 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d2d5749ac6a41b68188b1f8ffeef057" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 43971 } }
I20260926 15:10:16.311069 22810 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9d2d5749ac6a41b68188b1f8ffeef057 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 15:10:16.311197 22810 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9d2d5749ac6a41b68188b1f8ffeef057 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 15:10:16.311380 22810 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9d2d5749ac6a41b68188b1f8ffeef057 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:10:16.313539 22810 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9d2d5749ac6a41b68188b1f8ffeef057 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d2d5749ac6a41b68188b1f8ffeef057" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 43971 } }
I20260926 15:10:16.314002 22810 leader_election.cc:302] T 00000000000000000000000000000000 P 9d2d5749ac6a41b68188b1f8ffeef057 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9d2d5749ac6a41b68188b1f8ffeef057; no voters:
I20260926 15:10:16.314620 22810 leader_election.cc:288] T 00000000000000000000000000000000 P 9d2d5749ac6a41b68188b1f8ffeef057 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 15:10:16.314786 22815 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9d2d5749ac6a41b68188b1f8ffeef057 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:10:16.315493 22815 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9d2d5749ac6a41b68188b1f8ffeef057 [term 1 LEADER]: Becoming Leader. State: Replica: 9d2d5749ac6a41b68188b1f8ffeef057, State: Running, Role: LEADER
I20260926 15:10:16.316023 22815 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9d2d5749ac6a41b68188b1f8ffeef057 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d2d5749ac6a41b68188b1f8ffeef057" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 43971 } }
I20260926 15:10:16.316599 22810 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9d2d5749ac6a41b68188b1f8ffeef057 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:10:16.318660 22816 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d2d5749ac6a41b68188b1f8ffeef057 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9d2d5749ac6a41b68188b1f8ffeef057" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d2d5749ac6a41b68188b1f8ffeef057" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 43971 } } }
I20260926 15:10:16.319082 22816 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d2d5749ac6a41b68188b1f8ffeef057 [sys.catalog]: This master's current role is: LEADER
I20260926 15:10:16.319918 22821 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:10:16.320500 22817 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d2d5749ac6a41b68188b1f8ffeef057 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9d2d5749ac6a41b68188b1f8ffeef057. Latest consensus state: current_term: 1 leader_uuid: "9d2d5749ac6a41b68188b1f8ffeef057" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d2d5749ac6a41b68188b1f8ffeef057" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 43971 } } }
I20260926 15:10:16.320816 22817 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d2d5749ac6a41b68188b1f8ffeef057 [sys.catalog]: This master's current role is: LEADER
I20260926 15:10:16.325163 22821 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 15:10:16.332732 22821 catalog_manager.cc:1347] Generated new cluster ID: 64ad490fade943d7bb0285d52bdf61f1
I20260926 15:10:16.332826 22821 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 15:10:16.352293 22821 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 15:10:16.353624 22821 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:10:16.366848 22821 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9d2d5749ac6a41b68188b1f8ffeef057: Generated new TSK 0
I20260926 15:10:16.367758 22821 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.34.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.34.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.34.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.193.keytab.
Entry for principal kudu/127.12.34.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.193.keytab.
I20260926 15:10:16.440949 12427 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8uoJGv/build/asan/bin/kudu
/tmp/dist-test-task8uoJGv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435318278681-12427-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435318278681-12427-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435318278681-12427-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435318278681-12427-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.34.193:0
--local_ip_for_outbound_sockets=127.12.34.193
--webserver_interface=127.12.34.193
--webserver_port=0
--tserver_master_addrs=127.12.34.254:43971
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.193.keytab
--principal=kudu/127.12.34.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.34.212:44131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435318278681-12427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435318278681-12427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435318278681-12427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:10:16.779312 22836 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:10:16.779587 22836 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:10:16.779748 22836 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:10:16.786965 22836 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:10:16.787063 22836 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:10:16.787284 22836 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.34.193
I20260926 15:10:16.796908 22836 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.34.212:44131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435318278681-12427-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435318278681-12427-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.193.keytab
--principal=kudu/127.12.34.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.34.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435318278681-12427-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.34.193
--webserver_port=0
--tserver_master_addrs=127.12.34.254:43971
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.34.193
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435318278681-12427-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:10:16.798156 22836 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:10:16.799371 22836 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:10:16.812315 22841 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:10:16.812268 22842 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:10:16.812355 22844 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:10:16.812906 22836 server_base.cc:1034] running on GCE node
I20260926 15:10:16.813529 22836 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:10:16.814407 22836 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:10:16.815626 22836 hybrid_clock.cc:648] HybridClock initialized: now 1790435416815559 us; error 51 us; skew 500 ppm
I20260926 15:10:16.815985 22836 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:10:16 dist-test-slave-865v krb5kdc[22730](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435416, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:10:16.819555 22836 init.cc:377] Logged in from keytab as kudu/127.12.34.193@KRBTEST.COM (short username kudu)
I20260926 15:10:16.821275 22836 webserver.cc:460] Webserver started at http://127.12.34.193:43895/ using document root <none> and password file <none>
I20260926 15:10:16.822081 22836 fs_manager.cc:359] Metadata directory not provided
I20260926 15:10:16.822238 22836 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:10:16.822562 22836 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:10:16.825920 22836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435318278681-12427-0/minicluster-data/ts-0/data/instance:
uuid: "4e6a0f94c4b8447690ddf6f157dafc84"
format_stamp: "Formatted at 2026-09-26 15:10:16 on dist-test-slave-865v"
server_key: "b900fc551c7230a1b3a747e2919ab949"
server_key_iv: "e16b6e0477c15ed7d75e8becf203c1a8"
server_key_version: "encryptionkey@0"
I20260926 15:10:16.826762 22836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435318278681-12427-0/minicluster-data/ts-0/wal/instance:
uuid: "4e6a0f94c4b8447690ddf6f157dafc84"
format_stamp: "Formatted at 2026-09-26 15:10:16 on dist-test-slave-865v"
server_key: "b900fc551c7230a1b3a747e2919ab949"
server_key_iv: "e16b6e0477c15ed7d75e8becf203c1a8"
server_key_version: "encryptionkey@0"
I20260926 15:10:16.832249 22836 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 15:10:16.835737 22851 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:10:16.837201 22836 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 15:10:16.837376 22836 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435318278681-12427-0/minicluster-data/ts-0/data,/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435318278681-12427-0/minicluster-data/ts-0/wal
uuid: "4e6a0f94c4b8447690ddf6f157dafc84"
format_stamp: "Formatted at 2026-09-26 15:10:16 on dist-test-slave-865v"
server_key: "b900fc551c7230a1b3a747e2919ab949"
server_key_iv: "e16b6e0477c15ed7d75e8becf203c1a8"
server_key_version: "encryptionkey@0"
I20260926 15:10:16.837617 22836 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435318278681-12427-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435318278681-12427-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435318278681-12427-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:10:16.862995 22836 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:10:16.867233 22836 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:10:16.867530 22836 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:10:16.868453 22836 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:10:16.869791 22836 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:10:16.869863 22836 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:10:16.870019 22836 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:10:16.870100 22836 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:10:16.901916 22836 rpc_server.cc:307] RPC server started. Bound to: 127.12.34.193:34439
I20260926 15:10:16.901963 22963 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.34.193:34439 every 8 connection(s)
I20260926 15:10:16.903153 22836 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435318278681-12427-0/minicluster-data/ts-0/data/info.pb
I20260926 15:10:16.906863 12427 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8uoJGv/build/asan/bin/kudu as pid 22836
I20260926 15:10:16.907083 12427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435318278681-12427-0/minicluster-data/ts-0/wal/instance
I20260926 15:10:16.907712 12427 external_mini_cluster.cc:1428] Setting key 932ad67f36581a8b998d6dc8bbb09363
Sep 26 15:10:16 dist-test-slave-865v krb5kdc[22730](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435416, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.193@KRBTEST.COM for kudu/127.12.34.254@KRBTEST.COM
I20260926 15:10:16.922472 22966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:10:16.905606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.193:56115 (local address 127.12.34.254:43971)
0926 15:10:16.906016 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0926 15:10:16.906021 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:10:16.906928 (+ 907us) server_negotiation.cc:407] Connection header received
0926 15:10:16.908907 (+ 1979us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:10:16.908912 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:10:16.908999 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:10:16.909122 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:10:16.910578 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:10:16.911276 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:10:16.912734 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:10:16.913184 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:10:16.913317 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:10:16.916641 (+ 3324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:10:16.916650 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:10:16.916659 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:10:16.916690 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:10:16.919286 (+ 2596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:10:16.920169 (+ 883us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:10:16.920172 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:10:16.920175 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:10:16.920296 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:10:16.920762 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:10:16.920766 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:10:16.920768 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:10:16.921014 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:10:16.921206 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0926 15:10:16.921984 (+ 778us) server_negotiation.cc:299] Negotiation successful
0926 15:10:16.922143 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":219,"threads_started":1}
I20260926 15:10:16.923856 22967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:10:16.906148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:43971 (local address 127.12.34.193:56115)
0926 15:10:16.906782 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0926 15:10:16.906820 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 15:10:16.908668 (+ 1848us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:10:16.909406 (+ 738us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:10:16.909416 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:10:16.909909 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:10:16.910413 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:10:16.910429 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:10:16.911426 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:10:16.911437 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:10:16.912577 (+ 1140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:10:16.912592 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:10:16.913403 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:10:16.913412 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:10:16.913625 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:10:16.914282 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:10:16.914310 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:10:16.916417 (+ 2107us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:10:16.919423 (+ 3006us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:10:16.919434 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:10:16.919451 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:10:16.920050 (+ 599us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:10:16.920408 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:10:16.920416 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:10:16.920423 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:10:16.920654 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:10:16.921169 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:10:16.921180 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:10:16.921595 (+ 415us) client_negotiation.cc:769] Sending connection context
0926 15:10:16.921910 (+ 315us) client_negotiation.cc:240] Negotiation successful
0926 15:10:16.922455 (+ 545us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":262,"threads_started":1}
I20260926 15:10:16.925395 22964 heartbeater.cc:347] Connected to a master server at 127.12.34.254:43971
I20260926 15:10:16.925688 22964 heartbeater.cc:464] Registering TS with master...
I20260926 15:10:16.926443 22964 heartbeater.cc:511] Master 127.12.34.254:43971 requested a full tablet report, sending...
I20260926 15:10:16.928449 22775 ts_manager.cc:194] Registered new tserver with Master: 4e6a0f94c4b8447690ddf6f157dafc84 (127.12.34.193:34439)
WARNING: no policy specified for kudu/127.12.34.194@KRBTEST.COM; defaulting to no policy
I20260926 15:10:16.930037 22775 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.34.193@KRBTEST.COM'} at 127.12.34.193:56115
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.34.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.34.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.194.keytab.
Entry for principal kudu/127.12.34.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.194.keytab.
I20260926 15:10:16.960525 12427 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8uoJGv/build/asan/bin/kudu
/tmp/dist-test-task8uoJGv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435318278681-12427-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435318278681-12427-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435318278681-12427-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435318278681-12427-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.34.194:0
--local_ip_for_outbound_sockets=127.12.34.194
--webserver_interface=127.12.34.194
--webserver_port=0
--tserver_master_addrs=127.12.34.254:43971
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.194.keytab
--principal=kudu/127.12.34.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.34.212:44131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435318278681-12427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435318278681-12427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435318278681-12427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:10:17.276283 22970 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:10:17.276561 22970 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:10:17.276724 22970 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:10:17.283872 22970 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:10:17.284015 22970 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:10:17.284201 22970 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.34.194
I20260926 15:10:17.293591 22970 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.34.212:44131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435318278681-12427-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435318278681-12427-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.194.keytab
--principal=kudu/127.12.34.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.34.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435318278681-12427-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.34.194
--webserver_port=0
--tserver_master_addrs=127.12.34.254:43971
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.34.194
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435318278681-12427-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:10:17.294796 22970 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:10:17.296578 22970 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:10:17.310092 22976 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:10:17.310118 22978 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:10:17.310398 22975 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:10:17.311172 22970 server_base.cc:1034] running on GCE node
I20260926 15:10:17.311735 22970 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:10:17.312610 22970 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:10:17.313841 22970 hybrid_clock.cc:648] HybridClock initialized: now 1790435417313783 us; error 41 us; skew 500 ppm
I20260926 15:10:17.314205 22970 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:10:17 dist-test-slave-865v krb5kdc[22730](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435417, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:10:17.317431 22970 init.cc:377] Logged in from keytab as kudu/127.12.34.194@KRBTEST.COM (short username kudu)
I20260926 15:10:17.318902 22970 webserver.cc:460] Webserver started at http://127.12.34.194:36231/ using document root <none> and password file <none>
I20260926 15:10:17.319582 22970 fs_manager.cc:359] Metadata directory not provided
I20260926 15:10:17.319723 22970 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:10:17.320014 22970 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:10:17.323071 22970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435318278681-12427-0/minicluster-data/ts-1/data/instance:
uuid: "73ca249d1f7f47a197ba40b2ff72e962"
format_stamp: "Formatted at 2026-09-26 15:10:17 on dist-test-slave-865v"
server_key: "35d8fcfa54ee7ce3f52b56af512135ab"
server_key_iv: "018fd3bf94c1e06c2986e1590497323c"
server_key_version: "encryptionkey@0"
I20260926 15:10:17.323803 22970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435318278681-12427-0/minicluster-data/ts-1/wal/instance:
uuid: "73ca249d1f7f47a197ba40b2ff72e962"
format_stamp: "Formatted at 2026-09-26 15:10:17 on dist-test-slave-865v"
server_key: "35d8fcfa54ee7ce3f52b56af512135ab"
server_key_iv: "018fd3bf94c1e06c2986e1590497323c"
server_key_version: "encryptionkey@0"
I20260926 15:10:17.328732 22970 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260926 15:10:17.331944 22985 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:10:17.333204 22970 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 15:10:17.333379 22970 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435318278681-12427-0/minicluster-data/ts-1/data,/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435318278681-12427-0/minicluster-data/ts-1/wal
uuid: "73ca249d1f7f47a197ba40b2ff72e962"
format_stamp: "Formatted at 2026-09-26 15:10:17 on dist-test-slave-865v"
server_key: "35d8fcfa54ee7ce3f52b56af512135ab"
server_key_iv: "018fd3bf94c1e06c2986e1590497323c"
server_key_version: "encryptionkey@0"
I20260926 15:10:17.333572 22970 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435318278681-12427-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435318278681-12427-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435318278681-12427-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:10:17.354089 22970 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:10:17.358152 22970 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:10:17.358446 22970 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:10:17.359369 22970 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:10:17.360594 22970 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:10:17.360663 22970 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:10:17.360834 22970 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:10:17.360886 22970 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:10:17.391948 22970 rpc_server.cc:307] RPC server started. Bound to: 127.12.34.194:39981
I20260926 15:10:17.391983 23097 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.34.194:39981 every 8 connection(s)
I20260926 15:10:17.393177 22970 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435318278681-12427-0/minicluster-data/ts-1/data/info.pb
I20260926 15:10:17.402568 12427 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8uoJGv/build/asan/bin/kudu as pid 22970
I20260926 15:10:17.402770 12427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435318278681-12427-0/minicluster-data/ts-1/wal/instance
I20260926 15:10:17.403172 12427 external_mini_cluster.cc:1428] Setting key 1ff2d6d07ec456c9df017c857b0b1f81
Sep 26 15:10:17 dist-test-slave-865v krb5kdc[22730](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435417, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.194@KRBTEST.COM for kudu/127.12.34.254@KRBTEST.COM
I20260926 15:10:17.412134 22966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:10:17.395669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.194:54963 (local address 127.12.34.254:43971)
0926 15:10:17.395847 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0926 15:10:17.395852 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:10:17.396881 (+ 1029us) server_negotiation.cc:407] Connection header received
0926 15:10:17.398652 (+ 1771us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:10:17.398658 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:10:17.398750 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:10:17.398858 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:10:17.400164 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:10:17.400863 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:10:17.402371 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:10:17.402881 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:10:17.403008 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:10:17.406701 (+ 3693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:10:17.406706 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:10:17.406709 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:10:17.406739 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:10:17.409333 (+ 2594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:10:17.410008 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:10:17.410020 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:10:17.410023 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:10:17.410135 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:10:17.410567 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:10:17.410572 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:10:17.410575 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:10:17.410819 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:10:17.410964 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0926 15:10:17.411589 (+ 625us) server_negotiation.cc:299] Negotiation successful
0926 15:10:17.411803 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 15:10:17.413060 23100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:10:17.396092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:43971 (local address 127.12.34.194:54963)
0926 15:10:17.396741 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0926 15:10:17.396775 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 15:10:17.398408 (+ 1633us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:10:17.399072 (+ 664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:10:17.399079 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:10:17.399526 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:10:17.400009 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:10:17.400023 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:10:17.401007 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:10:17.401012 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:10:17.402231 (+ 1219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:10:17.402242 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:10:17.403091 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:10:17.403099 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:10:17.403369 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:10:17.404047 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:10:17.404073 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:10:17.406488 (+ 2415us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:10:17.409465 (+ 2977us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:10:17.409474 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:10:17.409489 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:10:17.409888 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:10:17.410250 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:10:17.410255 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:10:17.410259 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:10:17.410442 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:10:17.410941 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:10:17.410949 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:10:17.411257 (+ 308us) client_negotiation.cc:769] Sending connection context
0926 15:10:17.411497 (+ 240us) client_negotiation.cc:240] Negotiation successful
0926 15:10:17.412018 (+ 521us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":265,"threads_started":1}
I20260926 15:10:17.414529 23098 heartbeater.cc:347] Connected to a master server at 127.12.34.254:43971
I20260926 15:10:17.414811 23098 heartbeater.cc:464] Registering TS with master...
I20260926 15:10:17.415448 23098 heartbeater.cc:511] Master 127.12.34.254:43971 requested a full tablet report, sending...
I20260926 15:10:17.416882 22775 ts_manager.cc:194] Registered new tserver with Master: 73ca249d1f7f47a197ba40b2ff72e962 (127.12.34.194:39981)
I20260926 15:10:17.418138 22775 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.34.194@KRBTEST.COM'} at 127.12.34.194:54963
WARNING: no policy specified for kudu/127.12.34.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.34.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.34.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.195.keytab.
Entry for principal kudu/127.12.34.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.195.keytab.
I20260926 15:10:17.456661 12427 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8uoJGv/build/asan/bin/kudu
/tmp/dist-test-task8uoJGv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435318278681-12427-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435318278681-12427-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435318278681-12427-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435318278681-12427-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.34.195:0
--local_ip_for_outbound_sockets=127.12.34.195
--webserver_interface=127.12.34.195
--webserver_port=0
--tserver_master_addrs=127.12.34.254:43971
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.195.keytab
--principal=kudu/127.12.34.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.34.212:44131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435318278681-12427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435318278681-12427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435318278681-12427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:10:17.776191 23103 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:10:17.776443 23103 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:10:17.776559 23103 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:10:17.783614 23103 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:10:17.783710 23103 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:10:17.783814 23103 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.34.195
I20260926 15:10:17.793042 23103 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.34.212:44131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435318278681-12427-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435318278681-12427-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.195.keytab
--principal=kudu/127.12.34.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.34.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435318278681-12427-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.34.195
--webserver_port=0
--tserver_master_addrs=127.12.34.254:43971
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.34.195
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435318278681-12427-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:10:17.794245 23103 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:10:17.795415 23103 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:10:17.807652 23109 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:10:17.807777 23108 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:10:17.808100 23111 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:10:17.808739 23103 server_base.cc:1034] running on GCE node
I20260926 15:10:17.809376 23103 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:10:17.810201 23103 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:10:17.811419 23103 hybrid_clock.cc:648] HybridClock initialized: now 1790435417811393 us; error 66 us; skew 500 ppm
I20260926 15:10:17.811708 23103 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:10:17 dist-test-slave-865v krb5kdc[22730](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435417, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:10:17.814723 23103 init.cc:377] Logged in from keytab as kudu/127.12.34.195@KRBTEST.COM (short username kudu)
I20260926 15:10:17.816097 23103 webserver.cc:460] Webserver started at http://127.12.34.195:35939/ using document root <none> and password file <none>
I20260926 15:10:17.816762 23103 fs_manager.cc:359] Metadata directory not provided
I20260926 15:10:17.816857 23103 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:10:17.817168 23103 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:10:17.819850 23103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435318278681-12427-0/minicluster-data/ts-2/data/instance:
uuid: "6797de05a232483c9821125a9449292a"
format_stamp: "Formatted at 2026-09-26 15:10:17 on dist-test-slave-865v"
server_key: "f5ca5474813135d5c2c4e26937466c08"
server_key_iv: "bb0175111b020debf96e6720aaf75d06"
server_key_version: "encryptionkey@0"
I20260926 15:10:17.820513 23103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435318278681-12427-0/minicluster-data/ts-2/wal/instance:
uuid: "6797de05a232483c9821125a9449292a"
format_stamp: "Formatted at 2026-09-26 15:10:17 on dist-test-slave-865v"
server_key: "f5ca5474813135d5c2c4e26937466c08"
server_key_iv: "bb0175111b020debf96e6720aaf75d06"
server_key_version: "encryptionkey@0"
I20260926 15:10:17.825455 23103 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260926 15:10:17.828624 23118 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:10:17.830036 23103 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 15:10:17.830166 23103 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435318278681-12427-0/minicluster-data/ts-2/data,/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435318278681-12427-0/minicluster-data/ts-2/wal
uuid: "6797de05a232483c9821125a9449292a"
format_stamp: "Formatted at 2026-09-26 15:10:17 on dist-test-slave-865v"
server_key: "f5ca5474813135d5c2c4e26937466c08"
server_key_iv: "bb0175111b020debf96e6720aaf75d06"
server_key_version: "encryptionkey@0"
I20260926 15:10:17.830379 23103 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435318278681-12427-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435318278681-12427-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435318278681-12427-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:10:17.841039 23103 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:10:17.845405 23103 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:10:17.845638 23103 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:10:17.846442 23103 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:10:17.847611 23103 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:10:17.847709 23103 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:10:17.847826 23103 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:10:17.847872 23103 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:10:17.881017 23103 rpc_server.cc:307] RPC server started. Bound to: 127.12.34.195:44179
I20260926 15:10:17.881049 23230 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.34.195:44179 every 8 connection(s)
I20260926 15:10:17.882261 23103 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435318278681-12427-0/minicluster-data/ts-2/data/info.pb
I20260926 15:10:17.888592 12427 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8uoJGv/build/asan/bin/kudu as pid 23103
I20260926 15:10:17.888794 12427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435318278681-12427-0/minicluster-data/ts-2/wal/instance
I20260926 15:10:17.889315 12427 external_mini_cluster.cc:1428] Setting key dfe07e5eab1b1fffe8eec8431d6c4622
Sep 26 15:10:17 dist-test-slave-865v krb5kdc[22730](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435417, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.195@KRBTEST.COM for kudu/127.12.34.254@KRBTEST.COM
I20260926 15:10:17.900959 22966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:10:17.884614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.195:43165 (local address 127.12.34.254:43971)
0926 15:10:17.884793 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0926 15:10:17.884798 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:10:17.885807 (+ 1009us) server_negotiation.cc:407] Connection header received
0926 15:10:17.887532 (+ 1725us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:10:17.887537 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:10:17.887626 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:10:17.887736 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:10:17.889215 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:10:17.890111 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:10:17.891766 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:10:17.892157 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:10:17.892288 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:10:17.895598 (+ 3310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:10:17.895605 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:10:17.895610 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:10:17.895651 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:10:17.898120 (+ 2469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:10:17.898810 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:10:17.898815 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:10:17.898817 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:10:17.898931 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:10:17.899341 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:10:17.899347 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:10:17.899352 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:10:17.899615 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:10:17.899773 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0926 15:10:17.900410 (+ 637us) server_negotiation.cc:299] Negotiation successful
0926 15:10:17.900612 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 15:10:17.902050 23233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:10:17.884924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:43971 (local address 127.12.34.195:43165)
0926 15:10:17.885652 (+ 728us) negotiation.cc:107] Waiting for socket to connect
0926 15:10:17.885690 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 15:10:17.887291 (+ 1601us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:10:17.887945 (+ 654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:10:17.887952 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:10:17.888429 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:10:17.889028 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:10:17.889045 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:10:17.890250 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:10:17.890267 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:10:17.891609 (+ 1342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:10:17.891621 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:10:17.892374 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:10:17.892381 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:10:17.892594 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:10:17.893294 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:10:17.893315 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:10:17.895404 (+ 2089us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:10:17.898265 (+ 2861us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:10:17.898273 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:10:17.898287 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:10:17.898675 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:10:17.899037 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:10:17.899043 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:10:17.899047 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:10:17.899221 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:10:17.899746 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:10:17.899753 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:10:17.900054 (+ 301us) client_negotiation.cc:769] Sending connection context
0926 15:10:17.900312 (+ 258us) client_negotiation.cc:240] Negotiation successful
0926 15:10:17.900789 (+ 477us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":347,"threads_started":1}
I20260926 15:10:17.903601 23231 heartbeater.cc:347] Connected to a master server at 127.12.34.254:43971
I20260926 15:10:17.903944 23231 heartbeater.cc:464] Registering TS with master...
I20260926 15:10:17.904754 23231 heartbeater.cc:511] Master 127.12.34.254:43971 requested a full tablet report, sending...
I20260926 15:10:17.906282 22774 ts_manager.cc:194] Registered new tserver with Master: 6797de05a232483c9821125a9449292a (127.12.34.195:44179)
I20260926 15:10:17.907357 22774 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.34.195@KRBTEST.COM'} at 127.12.34.195:43165
I20260926 15:10:17.916610 12427 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 15:10:17.933568 22964 heartbeater.cc:503] Master 127.12.34.254:43971 was elected leader, sending a full tablet report...
I20260926 15:10:17.933717 22966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:10:17.921175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57050 (local address 127.12.34.254:43971)
0926 15:10:17.921375 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0926 15:10:17.921380 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:10:17.921880 (+ 500us) server_negotiation.cc:407] Connection header received
0926 15:10:17.922010 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:10:17.922013 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:10:17.922071 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:10:17.922155 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:10:17.923452 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:10:17.924166 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:10:17.925413 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:10:17.925867 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:10:17.925961 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:10:17.928239 (+ 2278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:10:17.928244 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:10:17.928248 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:10:17.928278 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:10:17.930998 (+ 2720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:10:17.931674 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:10:17.931678 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:10:17.931681 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:10:17.931819 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:10:17.932324 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:10:17.932334 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:10:17.932336 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:10:17.932670 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:10:17.932857 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0926 15:10:17.933204 (+ 347us) server_negotiation.cc:299] Negotiation successful
0926 15:10:17.933440 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260926 15:10:17.938184 22774 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57050:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 15:10:17.945436 22774 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 15:10:17.977277 23243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:10:17.957666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.193:34439 (local address 127.0.0.1:38292)
0926 15:10:17.967008 (+ 9342us) negotiation.cc:107] Waiting for socket to connect
0926 15:10:17.967065 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0926 15:10:17.967308 (+ 243us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:10:17.969039 (+ 1731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:10:17.969046 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:10:17.969102 (+ 56us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:10:17.969586 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:10:17.969597 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:10:17.971809 (+ 2212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:10:17.971816 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:10:17.974658 (+ 2842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:10:17.974672 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:10:17.976250 (+ 1578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:10:17.976257 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:10:17.976445 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:10:17.976500 (+ 55us) client_negotiation.cc:769] Sending connection context
0926 15:10:17.976637 (+ 137us) client_negotiation.cc:240] Negotiation successful
0926 15:10:17.976850 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8981,"mutex_wait_us":71,"thread_start_us":8788,"threads_started":1}
I20260926 15:10:17.977428 23244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:10:17.957934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38292 (local address 127.12.34.193:34439)
0926 15:10:17.968394 (+ 10460us) server_negotiation.cc:206] Beginning negotiation
0926 15:10:17.968403 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 15:10:17.968432 (+ 29us) server_negotiation.cc:407] Connection header received
0926 15:10:17.968535 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:10:17.968546 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:10:17.968824 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:10:17.969783 (+ 959us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:10:17.970014 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:10:17.971647 (+ 1633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:10:17.974848 (+ 3201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:10:17.976028 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:10:17.976182 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:10:17.976442 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0926 15:10:17.976900 (+ 458us) server_negotiation.cc:299] Negotiation successful
0926 15:10:17.977064 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10293,"thread_start_us":10127,"threads_started":1}
I20260926 15:10:17.977560 23246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:10:17.958642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48298 (local address 127.12.34.195:44179)
0926 15:10:17.968045 (+ 9403us) server_negotiation.cc:206] Beginning negotiation
0926 15:10:17.968057 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 15:10:17.968582 (+ 525us) server_negotiation.cc:407] Connection header received
0926 15:10:17.968856 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:10:17.968867 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:10:17.969067 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:10:17.969258 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:10:17.970236 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:10:17.971775 (+ 1539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:10:17.975096 (+ 3321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:10:17.976449 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:10:17.976578 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:10:17.976786 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0926 15:10:17.977016 (+ 230us) server_negotiation.cc:299] Negotiation successful
0926 15:10:17.977208 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9223,"thread_start_us":9042,"threads_started":1}
I20260926 15:10:17.978379 23245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:10:17.958050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.195:44179 (local address 127.0.0.1:48298)
0926 15:10:17.968481 (+ 10431us) negotiation.cc:107] Waiting for socket to connect
0926 15:10:17.968497 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 15:10:17.968673 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:10:17.969568 (+ 895us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:10:17.969573 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:10:17.969595 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:10:17.970016 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:10:17.970027 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:10:17.971941 (+ 1914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:10:17.971947 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:10:17.974927 (+ 2980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:10:17.974941 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:10:17.976637 (+ 1696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:10:17.976644 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:10:17.976813 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:10:17.976845 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 15:10:17.977735 (+ 890us) client_negotiation.cc:240] Negotiation successful
0926 15:10:17.977869 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10303,"mutex_wait_us":103,"thread_start_us":10156,"threads_started":1}
I20260926 15:10:17.982301 23248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:10:17.969603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.194:39981 (local address 127.0.0.1:54320)
0926 15:10:17.973108 (+ 3505us) negotiation.cc:107] Waiting for socket to connect
0926 15:10:17.973129 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 15:10:17.973242 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:10:17.975715 (+ 2473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:10:17.975722 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:10:17.975751 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:10:17.976128 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:10:17.976142 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:10:17.977975 (+ 1833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:10:17.977983 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:10:17.980001 (+ 2018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:10:17.980015 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:10:17.981536 (+ 1521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:10:17.981543 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:10:17.981701 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:10:17.981734 (+ 33us) client_negotiation.cc:769] Sending connection context
0926 15:10:17.981849 (+ 115us) client_negotiation.cc:240] Negotiation successful
0926 15:10:17.981968 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3368,"thread_start_us":3228,"threads_started":1}
I20260926 15:10:17.982336 23247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:10:17.968843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54320 (local address 127.12.34.194:39981)
0926 15:10:17.975157 (+ 6314us) server_negotiation.cc:206] Beginning negotiation
0926 15:10:17.975168 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 15:10:17.975195 (+ 27us) server_negotiation.cc:407] Connection header received
0926 15:10:17.975283 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:10:17.975291 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:10:17.975507 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:10:17.975695 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:10:17.976311 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:10:17.977830 (+ 1519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:10:17.980149 (+ 2319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:10:17.981339 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:10:17.981493 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:10:17.981681 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0926 15:10:17.981889 (+ 208us) server_negotiation.cc:299] Negotiation successful
0926 15:10:17.982033 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6148,"thread_start_us":6018,"threads_started":1}
I20260926 15:10:17.981935 22899 tablet_service.cc:1467] Processing CreateTablet for tablet 978d299b15304b59b306cdd181758dc9 (DEFAULT_TABLE table=test-table [id=e39546a4ae1845ebbd9bdd563b183a1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:10:17.981933 23166 tablet_service.cc:1467] Processing CreateTablet for tablet 978d299b15304b59b306cdd181758dc9 (DEFAULT_TABLE table=test-table [id=e39546a4ae1845ebbd9bdd563b183a1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:10:17.983415 22899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 978d299b15304b59b306cdd181758dc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:10:17.983409 23166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 978d299b15304b59b306cdd181758dc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:10:17.985364 23033 tablet_service.cc:1467] Processing CreateTablet for tablet 978d299b15304b59b306cdd181758dc9 (DEFAULT_TABLE table=test-table [id=e39546a4ae1845ebbd9bdd563b183a1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:10:17.986404 23033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 978d299b15304b59b306cdd181758dc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:10:17.997499 23249 tablet_bootstrap.cc:492] T 978d299b15304b59b306cdd181758dc9 P 73ca249d1f7f47a197ba40b2ff72e962: Bootstrap starting.
I20260926 15:10:18.001046 23249 tablet_bootstrap.cc:654] T 978d299b15304b59b306cdd181758dc9 P 73ca249d1f7f47a197ba40b2ff72e962: Neither blocks nor log segments found. Creating new log.
I20260926 15:10:18.001178 23251 tablet_bootstrap.cc:492] T 978d299b15304b59b306cdd181758dc9 P 6797de05a232483c9821125a9449292a: Bootstrap starting.
I20260926 15:10:18.002441 23249 log.cc:824] T 978d299b15304b59b306cdd181758dc9 P 73ca249d1f7f47a197ba40b2ff72e962: Log is configured to *not* fsync() on all Append() calls
I20260926 15:10:18.003067 23250 tablet_bootstrap.cc:492] T 978d299b15304b59b306cdd181758dc9 P 4e6a0f94c4b8447690ddf6f157dafc84: Bootstrap starting.
I20260926 15:10:18.004736 23251 tablet_bootstrap.cc:654] T 978d299b15304b59b306cdd181758dc9 P 6797de05a232483c9821125a9449292a: Neither blocks nor log segments found. Creating new log.
I20260926 15:10:18.005795 23251 log.cc:824] T 978d299b15304b59b306cdd181758dc9 P 6797de05a232483c9821125a9449292a: Log is configured to *not* fsync() on all Append() calls
I20260926 15:10:18.007298 23250 tablet_bootstrap.cc:654] T 978d299b15304b59b306cdd181758dc9 P 4e6a0f94c4b8447690ddf6f157dafc84: Neither blocks nor log segments found. Creating new log.
I20260926 15:10:18.008658 23250 log.cc:824] T 978d299b15304b59b306cdd181758dc9 P 4e6a0f94c4b8447690ddf6f157dafc84: Log is configured to *not* fsync() on all Append() calls
I20260926 15:10:18.010200 23251 tablet_bootstrap.cc:492] T 978d299b15304b59b306cdd181758dc9 P 6797de05a232483c9821125a9449292a: No bootstrap required, opened a new log
I20260926 15:10:18.010259 23249 tablet_bootstrap.cc:492] T 978d299b15304b59b306cdd181758dc9 P 73ca249d1f7f47a197ba40b2ff72e962: No bootstrap required, opened a new log
I20260926 15:10:18.010476 23251 ts_tablet_manager.cc:1397] T 978d299b15304b59b306cdd181758dc9 P 6797de05a232483c9821125a9449292a: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260926 15:10:18.010576 23249 ts_tablet_manager.cc:1397] T 978d299b15304b59b306cdd181758dc9 P 73ca249d1f7f47a197ba40b2ff72e962: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.007s
I20260926 15:10:18.011523 23250 tablet_bootstrap.cc:492] T 978d299b15304b59b306cdd181758dc9 P 4e6a0f94c4b8447690ddf6f157dafc84: No bootstrap required, opened a new log
I20260926 15:10:18.011775 23250 ts_tablet_manager.cc:1397] T 978d299b15304b59b306cdd181758dc9 P 4e6a0f94c4b8447690ddf6f157dafc84: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.007s
I20260926 15:10:18.017494 23249 raft_consensus.cc:348] T 978d299b15304b59b306cdd181758dc9 P 73ca249d1f7f47a197ba40b2ff72e962 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e6a0f94c4b8447690ddf6f157dafc84" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 34439 } } peers { permanent_uuid: "6797de05a232483c9821125a9449292a" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 44179 } } peers { permanent_uuid: "73ca249d1f7f47a197ba40b2ff72e962" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 39981 } }
I20260926 15:10:18.017907 23249 raft_consensus.cc:374] T 978d299b15304b59b306cdd181758dc9 P 73ca249d1f7f47a197ba40b2ff72e962 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:10:18.018035 23249 raft_consensus.cc:729] T 978d299b15304b59b306cdd181758dc9 P 73ca249d1f7f47a197ba40b2ff72e962 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73ca249d1f7f47a197ba40b2ff72e962, State: Initialized, Role: FOLLOWER
I20260926 15:10:18.018733 23249 consensus_queue.cc:260] T 978d299b15304b59b306cdd181758dc9 P 73ca249d1f7f47a197ba40b2ff72e962 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e6a0f94c4b8447690ddf6f157dafc84" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 34439 } } peers { permanent_uuid: "6797de05a232483c9821125a9449292a" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 44179 } } peers { permanent_uuid: "73ca249d1f7f47a197ba40b2ff72e962" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 39981 } }
I20260926 15:10:18.020646 23249 ts_tablet_manager.cc:1428] T 978d299b15304b59b306cdd181758dc9 P 73ca249d1f7f47a197ba40b2ff72e962: Time spent starting tablet: real 0.010s user 0.009s sys 0.000s
I20260926 15:10:18.020885 23098 heartbeater.cc:503] Master 127.12.34.254:43971 was elected leader, sending a full tablet report...
I20260926 15:10:18.020851 23251 raft_consensus.cc:348] T 978d299b15304b59b306cdd181758dc9 P 6797de05a232483c9821125a9449292a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e6a0f94c4b8447690ddf6f157dafc84" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 34439 } } peers { permanent_uuid: "6797de05a232483c9821125a9449292a" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 44179 } } peers { permanent_uuid: "73ca249d1f7f47a197ba40b2ff72e962" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 39981 } }
I20260926 15:10:18.021284 23251 raft_consensus.cc:374] T 978d299b15304b59b306cdd181758dc9 P 6797de05a232483c9821125a9449292a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:10:18.021381 23251 raft_consensus.cc:729] T 978d299b15304b59b306cdd181758dc9 P 6797de05a232483c9821125a9449292a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6797de05a232483c9821125a9449292a, State: Initialized, Role: FOLLOWER
I20260926 15:10:18.021919 23251 consensus_queue.cc:260] T 978d299b15304b59b306cdd181758dc9 P 6797de05a232483c9821125a9449292a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e6a0f94c4b8447690ddf6f157dafc84" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 34439 } } peers { permanent_uuid: "6797de05a232483c9821125a9449292a" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 44179 } } peers { permanent_uuid: "73ca249d1f7f47a197ba40b2ff72e962" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 39981 } }
I20260926 15:10:18.022083 23250 raft_consensus.cc:348] T 978d299b15304b59b306cdd181758dc9 P 4e6a0f94c4b8447690ddf6f157dafc84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e6a0f94c4b8447690ddf6f157dafc84" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 34439 } } peers { permanent_uuid: "6797de05a232483c9821125a9449292a" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 44179 } } peers { permanent_uuid: "73ca249d1f7f47a197ba40b2ff72e962" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 39981 } }
I20260926 15:10:18.022495 23250 raft_consensus.cc:374] T 978d299b15304b59b306cdd181758dc9 P 4e6a0f94c4b8447690ddf6f157dafc84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:10:18.022609 23250 raft_consensus.cc:729] T 978d299b15304b59b306cdd181758dc9 P 4e6a0f94c4b8447690ddf6f157dafc84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e6a0f94c4b8447690ddf6f157dafc84, State: Initialized, Role: FOLLOWER
I20260926 15:10:18.023284 23250 consensus_queue.cc:260] T 978d299b15304b59b306cdd181758dc9 P 4e6a0f94c4b8447690ddf6f157dafc84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e6a0f94c4b8447690ddf6f157dafc84" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 34439 } } peers { permanent_uuid: "6797de05a232483c9821125a9449292a" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 44179 } } peers { permanent_uuid: "73ca249d1f7f47a197ba40b2ff72e962" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 39981 } }
I20260926 15:10:18.025252 23250 ts_tablet_manager.cc:1428] T 978d299b15304b59b306cdd181758dc9 P 4e6a0f94c4b8447690ddf6f157dafc84: Time spent starting tablet: real 0.013s user 0.008s sys 0.007s
I20260926 15:10:18.026055 23231 heartbeater.cc:503] Master 127.12.34.254:43971 was elected leader, sending a full tablet report...
I20260926 15:10:18.026649 23251 ts_tablet_manager.cc:1428] T 978d299b15304b59b306cdd181758dc9 P 6797de05a232483c9821125a9449292a: Time spent starting tablet: real 0.016s user 0.010s sys 0.007s
I20260926 15:10:18.048864 23256 raft_consensus.cc:482] T 978d299b15304b59b306cdd181758dc9 P 6797de05a232483c9821125a9449292a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 15:10:18.049069 23256 raft_consensus.cc:504] T 978d299b15304b59b306cdd181758dc9 P 6797de05a232483c9821125a9449292a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e6a0f94c4b8447690ddf6f157dafc84" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 34439 } } peers { permanent_uuid: "6797de05a232483c9821125a9449292a" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 44179 } } peers { permanent_uuid: "73ca249d1f7f47a197ba40b2ff72e962" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 39981 } }
I20260926 15:10:18.050518 23256 leader_election.cc:288] T 978d299b15304b59b306cdd181758dc9 P 6797de05a232483c9821125a9449292a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4e6a0f94c4b8447690ddf6f157dafc84 (127.12.34.193:34439), 73ca249d1f7f47a197ba40b2ff72e962 (127.12.34.194:39981)
I20260926 15:10:18.056493 23233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:10:18.050773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.194:39981 (local address 127.12.34.195:57685)
0926 15:10:18.051087 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0926 15:10:18.051108 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 15:10:18.051244 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:10:18.051650 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:10:18.051653 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:10:18.051681 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:10:18.051852 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:10:18.051859 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:10:18.053089 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:10:18.053095 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:10:18.054655 (+ 1560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:10:18.054669 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:10:18.055982 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:10:18.055986 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:10:18.056110 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:10:18.056133 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 15:10:18.056209 (+ 76us) client_negotiation.cc:240] Negotiation successful
0926 15:10:18.056309 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163}
I20260926 15:10:18.056629 23247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:10:18.051001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.195:57685 (local address 127.12.34.194:39981)
0926 15:10:18.051185 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0926 15:10:18.051191 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:10:18.051212 (+ 21us) server_negotiation.cc:407] Connection header received
0926 15:10:18.051430 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:10:18.051438 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:10:18.051520 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:10:18.051618 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:10:18.051977 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:10:18.052934 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:10:18.054813 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:10:18.055845 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:10:18.055996 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:10:18.056062 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 15:10:18.056254 (+ 192us) server_negotiation.cc:299] Negotiation successful
0926 15:10:18.056383 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 15:10:18.057530 23244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:10:18.051008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.195:46663 (local address 127.12.34.193:34439)
0926 15:10:18.051185 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0926 15:10:18.051191 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:10:18.051824 (+ 633us) server_negotiation.cc:407] Connection header received
0926 15:10:18.052042 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:10:18.052046 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:10:18.052099 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:10:18.052164 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:10:18.052680 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:10:18.053651 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:10:18.055481 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:10:18.056666 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:10:18.056841 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:10:18.056915 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 15:10:18.057184 (+ 269us) server_negotiation.cc:299] Negotiation successful
0926 15:10:18.057304 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 15:10:18.057605 23258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:10:18.050892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.193:34439 (local address 127.12.34.195:46663)
0926 15:10:18.051700 (+ 808us) negotiation.cc:107] Waiting for socket to connect
0926 15:10:18.051731 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0926 15:10:18.051879 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:10:18.052213 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:10:18.052218 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:10:18.052243 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:10:18.052539 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:10:18.052549 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:10:18.053799 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:10:18.053802 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:10:18.055312 (+ 1510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:10:18.055325 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:10:18.056862 (+ 1537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:10:18.056868 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:10:18.056997 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:10:18.057020 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 15:10:18.057156 (+ 136us) client_negotiation.cc:240] Negotiation successful
0926 15:10:18.057289 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":693,"thread_start_us":486,"threads_started":1}
I20260926 15:10:18.057471 23053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "978d299b15304b59b306cdd181758dc9" candidate_uuid: "6797de05a232483c9821125a9449292a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73ca249d1f7f47a197ba40b2ff72e962" is_pre_election: true
I20260926 15:10:18.057942 23053 raft_consensus.cc:2454] T 978d299b15304b59b306cdd181758dc9 P 73ca249d1f7f47a197ba40b2ff72e962 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6797de05a232483c9821125a9449292a in term 0.
I20260926 15:10:18.058638 23122 leader_election.cc:302] T 978d299b15304b59b306cdd181758dc9 P 6797de05a232483c9821125a9449292a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6797de05a232483c9821125a9449292a, 73ca249d1f7f47a197ba40b2ff72e962; no voters:
I20260926 15:10:18.058616 22919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "978d299b15304b59b306cdd181758dc9" candidate_uuid: "6797de05a232483c9821125a9449292a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e6a0f94c4b8447690ddf6f157dafc84" is_pre_election: true
I20260926 15:10:18.059010 23256 raft_consensus.cc:2781] T 978d299b15304b59b306cdd181758dc9 P 6797de05a232483c9821125a9449292a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 15:10:18.059120 23256 raft_consensus.cc:482] T 978d299b15304b59b306cdd181758dc9 P 6797de05a232483c9821125a9449292a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 15:10:18.059134 22919 raft_consensus.cc:2454] T 978d299b15304b59b306cdd181758dc9 P 4e6a0f94c4b8447690ddf6f157dafc84 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6797de05a232483c9821125a9449292a in term 0.
I20260926 15:10:18.059260 23256 raft_consensus.cc:3037] T 978d299b15304b59b306cdd181758dc9 P 6797de05a232483c9821125a9449292a [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:10:18.061523 23256 raft_consensus.cc:504] T 978d299b15304b59b306cdd181758dc9 P 6797de05a232483c9821125a9449292a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e6a0f94c4b8447690ddf6f157dafc84" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 34439 } } peers { permanent_uuid: "6797de05a232483c9821125a9449292a" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 44179 } } peers { permanent_uuid: "73ca249d1f7f47a197ba40b2ff72e962" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 39981 } }
I20260926 15:10:18.062049 23256 leader_election.cc:288] T 978d299b15304b59b306cdd181758dc9 P 6797de05a232483c9821125a9449292a [CANDIDATE]: Term 1 election: Requested vote from peers 4e6a0f94c4b8447690ddf6f157dafc84 (127.12.34.193:34439), 73ca249d1f7f47a197ba40b2ff72e962 (127.12.34.194:39981)
I20260926 15:10:18.062619 23053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "978d299b15304b59b306cdd181758dc9" candidate_uuid: "6797de05a232483c9821125a9449292a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73ca249d1f7f47a197ba40b2ff72e962"
I20260926 15:10:18.062619 22919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "978d299b15304b59b306cdd181758dc9" candidate_uuid: "6797de05a232483c9821125a9449292a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e6a0f94c4b8447690ddf6f157dafc84"
I20260926 15:10:18.062814 22919 raft_consensus.cc:3037] T 978d299b15304b59b306cdd181758dc9 P 4e6a0f94c4b8447690ddf6f157dafc84 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:10:18.062814 23053 raft_consensus.cc:3037] T 978d299b15304b59b306cdd181758dc9 P 73ca249d1f7f47a197ba40b2ff72e962 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:10:18.065919 23053 raft_consensus.cc:2454] T 978d299b15304b59b306cdd181758dc9 P 73ca249d1f7f47a197ba40b2ff72e962 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6797de05a232483c9821125a9449292a in term 1.
I20260926 15:10:18.065919 22919 raft_consensus.cc:2454] T 978d299b15304b59b306cdd181758dc9 P 4e6a0f94c4b8447690ddf6f157dafc84 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6797de05a232483c9821125a9449292a in term 1.
I20260926 15:10:18.066345 23119 leader_election.cc:302] T 978d299b15304b59b306cdd181758dc9 P 6797de05a232483c9821125a9449292a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e6a0f94c4b8447690ddf6f157dafc84, 6797de05a232483c9821125a9449292a; no voters:
I20260926 15:10:18.066757 23256 raft_consensus.cc:2781] T 978d299b15304b59b306cdd181758dc9 P 6797de05a232483c9821125a9449292a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:10:18.067238 23256 raft_consensus.cc:686] T 978d299b15304b59b306cdd181758dc9 P 6797de05a232483c9821125a9449292a [term 1 LEADER]: Becoming Leader. State: Replica: 6797de05a232483c9821125a9449292a, State: Running, Role: LEADER
I20260926 15:10:18.067689 23256 consensus_queue.cc:237] T 978d299b15304b59b306cdd181758dc9 P 6797de05a232483c9821125a9449292a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e6a0f94c4b8447690ddf6f157dafc84" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 34439 } } peers { permanent_uuid: "6797de05a232483c9821125a9449292a" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 44179 } } peers { permanent_uuid: "73ca249d1f7f47a197ba40b2ff72e962" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 39981 } }
I20260926 15:10:18.072284 22775 catalog_manager.cc:5495] T 978d299b15304b59b306cdd181758dc9 P 6797de05a232483c9821125a9449292a reported cstate change: term changed from 0 to 1, leader changed from <none> to 6797de05a232483c9821125a9449292a (127.12.34.195). New cstate: current_term: 1 leader_uuid: "6797de05a232483c9821125a9449292a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e6a0f94c4b8447690ddf6f157dafc84" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 34439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6797de05a232483c9821125a9449292a" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 44179 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "73ca249d1f7f47a197ba40b2ff72e962" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 39981 } health_report { overall_health: UNKNOWN } } }
I20260926 15:10:18.090912 23246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:10:18.085644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48314 (local address 127.12.34.195:44179)
0926 15:10:18.085814 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0926 15:10:18.085818 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:10:18.085878 (+ 60us) server_negotiation.cc:407] Connection header received
0926 15:10:18.086033 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:10:18.086037 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:10:18.086117 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:10:18.086236 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 15:10:18.086525 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:10:18.087226 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:10:18.088515 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:10:18.088933 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:10:18.089038 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:10:18.089768 (+ 730us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 15:10:18.090273 (+ 505us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 15:10:18.090386 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0926 15:10:18.090546 (+ 160us) server_negotiation.cc:299] Negotiation successful
0926 15:10:18.090613 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 15:10:18.099072 22919 raft_consensus.cc:1261] T 978d299b15304b59b306cdd181758dc9 P 4e6a0f94c4b8447690ddf6f157dafc84 [term 1 FOLLOWER]: Refusing update from remote peer 6797de05a232483c9821125a9449292a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:10:18.099072 23053 raft_consensus.cc:1261] T 978d299b15304b59b306cdd181758dc9 P 73ca249d1f7f47a197ba40b2ff72e962 [term 1 FOLLOWER]: Refusing update from remote peer 6797de05a232483c9821125a9449292a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:10:18.100205 23259 consensus_queue.cc:1035] T 978d299b15304b59b306cdd181758dc9 P 6797de05a232483c9821125a9449292a [LEADER]: Connected to new peer: Peer: permanent_uuid: "73ca249d1f7f47a197ba40b2ff72e962" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 39981 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:10:18.100653 23256 consensus_queue.cc:1035] T 978d299b15304b59b306cdd181758dc9 P 6797de05a232483c9821125a9449292a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e6a0f94c4b8447690ddf6f157dafc84" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 34439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:10:18.111326 23262 mvcc.cc:205] Tried to move back new op lower bound from 7333623472523472896 to 7333623472409100288. Current Snapshot: MvccSnapshot[applied={T|T < 7333623472523472896}]
I20260926 15:10:18.118991 23264 mvcc.cc:205] Tried to move back new op lower bound from 7333623472523472896 to 7333623472409100288. Current Snapshot: MvccSnapshot[applied={T|T < 7333623472523472896}]
I20260926 15:10:18.125731 22774 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57050:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 15:10:18.125990 22774 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57050:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 15:10:18.129310 22774 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9d2d5749ac6a41b68188b1f8ffeef057: Sending DeleteTablet for 3 replicas of tablet 978d299b15304b59b306cdd181758dc9
I20260926 15:10:18.130375 22899 tablet_service.cc:1514] Processing DeleteTablet for tablet 978d299b15304b59b306cdd181758dc9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:10:18 UTC) from {username='kudu', principal='kudu/127.12.34.254@KRBTEST.COM'} at 127.0.0.1:38292
I20260926 15:10:18.130441 23033 tablet_service.cc:1514] Processing DeleteTablet for tablet 978d299b15304b59b306cdd181758dc9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:10:18 UTC) from {username='kudu', principal='kudu/127.12.34.254@KRBTEST.COM'} at 127.0.0.1:54320
I20260926 15:10:18.130441 23166 tablet_service.cc:1514] Processing DeleteTablet for tablet 978d299b15304b59b306cdd181758dc9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:10:18 UTC) from {username='kudu', principal='kudu/127.12.34.254@KRBTEST.COM'} at 127.0.0.1:48298
I20260926 15:10:18.131346 23272 tablet_replica.cc:330] T 978d299b15304b59b306cdd181758dc9 P 73ca249d1f7f47a197ba40b2ff72e962: stopping tablet replica
I20260926 15:10:18.131526 23271 tablet_replica.cc:330] T 978d299b15304b59b306cdd181758dc9 P 4e6a0f94c4b8447690ddf6f157dafc84: stopping tablet replica
I20260926 15:10:18.131929 23273 tablet_replica.cc:330] T 978d299b15304b59b306cdd181758dc9 P 6797de05a232483c9821125a9449292a: stopping tablet replica
I20260926 15:10:18.131958 23271 raft_consensus.cc:2229] T 978d299b15304b59b306cdd181758dc9 P 4e6a0f94c4b8447690ddf6f157dafc84 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 15:10:18.132229 12427 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 22836
I20260926 15:10:18.132581 23271 raft_consensus.cc:2258] T 978d299b15304b59b306cdd181758dc9 P 4e6a0f94c4b8447690ddf6f157dafc84 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:10:18.132860 23273 raft_consensus.cc:2229] T 978d299b15304b59b306cdd181758dc9 P 6797de05a232483c9821125a9449292a [term 1 LEADER]: Raft consensus shutting down.
I20260926 15:10:18.133126 23272 raft_consensus.cc:2229] T 978d299b15304b59b306cdd181758dc9 P 73ca249d1f7f47a197ba40b2ff72e962 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 15:10:18.133999 23272 raft_consensus.cc:2258] T 978d299b15304b59b306cdd181758dc9 P 73ca249d1f7f47a197ba40b2ff72e962 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:10:18.134922 23271 ts_tablet_manager.cc:1905] T 978d299b15304b59b306cdd181758dc9 P 4e6a0f94c4b8447690ddf6f157dafc84: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:10:18.134300 23273 raft_consensus.cc:2258] T 978d299b15304b59b306cdd181758dc9 P 6797de05a232483c9821125a9449292a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:10:18.136662 23272 ts_tablet_manager.cc:1905] T 978d299b15304b59b306cdd181758dc9 P 73ca249d1f7f47a197ba40b2ff72e962: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:10:18.138453 23273 ts_tablet_manager.cc:1905] T 978d299b15304b59b306cdd181758dc9 P 6797de05a232483c9821125a9449292a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:10:18.141959 23271 ts_tablet_manager.cc:1918] T 978d299b15304b59b306cdd181758dc9 P 4e6a0f94c4b8447690ddf6f157dafc84: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 15:10:18.142124 23271 log.cc:1196] T 978d299b15304b59b306cdd181758dc9 P 4e6a0f94c4b8447690ddf6f157dafc84: Deleting WAL directory at /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435318278681-12427-0/minicluster-data/ts-0/wal/wals/978d299b15304b59b306cdd181758dc9
I20260926 15:10:18.142647 23271 ts_tablet_manager.cc:1939] T 978d299b15304b59b306cdd181758dc9 P 4e6a0f94c4b8447690ddf6f157dafc84: Deleting consensus metadata
I20260926 15:10:18.144371 22759 catalog_manager.cc:4841] TS 4e6a0f94c4b8447690ddf6f157dafc84 (127.12.34.193:34439): tablet 978d299b15304b59b306cdd181758dc9 (table test-table [id=e39546a4ae1845ebbd9bdd563b183a1b]) successfully deleted
I20260926 15:10:18.145022 23273 ts_tablet_manager.cc:1918] T 978d299b15304b59b306cdd181758dc9 P 6797de05a232483c9821125a9449292a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 15:10:18.145196 23273 log.cc:1196] T 978d299b15304b59b306cdd181758dc9 P 6797de05a232483c9821125a9449292a: Deleting WAL directory at /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435318278681-12427-0/minicluster-data/ts-2/wal/wals/978d299b15304b59b306cdd181758dc9
I20260926 15:10:18.145536 23272 ts_tablet_manager.cc:1918] T 978d299b15304b59b306cdd181758dc9 P 73ca249d1f7f47a197ba40b2ff72e962: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 15:10:18.145699 23273 ts_tablet_manager.cc:1939] T 978d299b15304b59b306cdd181758dc9 P 6797de05a232483c9821125a9449292a: Deleting consensus metadata
I20260926 15:10:18.145721 23272 log.cc:1196] T 978d299b15304b59b306cdd181758dc9 P 73ca249d1f7f47a197ba40b2ff72e962: Deleting WAL directory at /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790435318278681-12427-0/minicluster-data/ts-1/wal/wals/978d299b15304b59b306cdd181758dc9
I20260926 15:10:18.146332 23272 ts_tablet_manager.cc:1939] T 978d299b15304b59b306cdd181758dc9 P 73ca249d1f7f47a197ba40b2ff72e962: Deleting consensus metadata
I20260926 15:10:18.147259 22762 catalog_manager.cc:4841] TS 6797de05a232483c9821125a9449292a (127.12.34.195:44179): tablet 978d299b15304b59b306cdd181758dc9 (table test-table [id=e39546a4ae1845ebbd9bdd563b183a1b]) successfully deleted
I20260926 15:10:18.147912 22762 catalog_manager.cc:4841] TS 73ca249d1f7f47a197ba40b2ff72e962 (127.12.34.194:39981): tablet 978d299b15304b59b306cdd181758dc9 (table test-table [id=e39546a4ae1845ebbd9bdd563b183a1b]) successfully deleted
Sep 26 15:10:18 dist-test-slave-865v krb5kdc[22730](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435415, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.34.193@KRBTEST.COM
I20260926 15:10:18.159821 23244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:10:18.134432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38302 (local address 127.12.34.193:34439)
0926 15:10:18.134586 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0926 15:10:18.134593 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:10:18.143452 (+ 8859us) server_negotiation.cc:407] Connection header received
0926 15:10:18.143888 (+ 436us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:10:18.143893 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:10:18.143972 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:10:18.144084 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:10:18.145982 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:10:18.147213 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:10:18.148721 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:10:18.149377 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:10:18.149508 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:10:18.154356 (+ 4848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:10:18.154366 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:10:18.154379 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:10:18.154413 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:10:18.157217 (+ 2804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:10:18.157953 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:10:18.157959 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:10:18.157961 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:10:18.158070 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:10:18.158483 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:10:18.158486 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:10:18.158488 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:10:18.158803 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:10:18.159009 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0926 15:10:18.159395 (+ 386us) server_negotiation.cc:299] Negotiation successful
0926 15:10:18.159535 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 15:10:18.160422 22959 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:10:18.288275 12427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8uoJGv/build/asan/bin/kudu with pid 22836
I20260926 15:10:18.308816 12427 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 22970
Sep 26 15:10:18 dist-test-slave-865v krb5kdc[22730](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435415, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.34.194@KRBTEST.COM
I20260926 15:10:18.327075 23247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:10:18.309661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54324 (local address 127.12.34.194:39981)
0926 15:10:18.309847 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0926 15:10:18.309851 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:10:18.309873 (+ 22us) server_negotiation.cc:407] Connection header received
0926 15:10:18.310165 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:10:18.310170 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:10:18.310260 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:10:18.310360 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:10:18.311925 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:10:18.312719 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:10:18.314565 (+ 1846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:10:18.314950 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:10:18.315054 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:10:18.319951 (+ 4897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:10:18.319970 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:10:18.319991 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:10:18.320033 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:10:18.323827 (+ 3794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:10:18.324663 (+ 836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:10:18.324674 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:10:18.324677 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:10:18.324890 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:10:18.325411 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:10:18.325417 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:10:18.325422 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:10:18.325892 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:10:18.326183 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0926 15:10:18.326486 (+ 303us) server_negotiation.cc:299] Negotiation successful
0926 15:10:18.326702 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260926 15:10:18.327797 23093 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:10:18.421290 12427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8uoJGv/build/asan/bin/kudu with pid 22970
I20260926 15:10:18.438933 12427 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 23103
Sep 26 15:10:18 dist-test-slave-865v krb5kdc[22730](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435415, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.34.195@KRBTEST.COM
I20260926 15:10:18.457535 23246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:10:18.439781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48326 (local address 127.12.34.195:44179)
0926 15:10:18.439954 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0926 15:10:18.439966 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 15:10:18.439992 (+ 26us) server_negotiation.cc:407] Connection header received
0926 15:10:18.440258 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:10:18.440263 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:10:18.440351 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:10:18.440439 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:10:18.442217 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:10:18.443013 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:10:18.444841 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:10:18.445449 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:10:18.445561 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:10:18.451332 (+ 5771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:10:18.451347 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:10:18.451366 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:10:18.451412 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:10:18.454546 (+ 3134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:10:18.455290 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:10:18.455300 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:10:18.455302 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:10:18.455463 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:10:18.455866 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:10:18.455871 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:10:18.455884 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:10:18.456225 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:10:18.456545 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0926 15:10:18.456826 (+ 281us) server_negotiation.cc:299] Negotiation successful
0926 15:10:18.457063 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 15:10:18.458187 23226 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:10:18.580412 12427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8uoJGv/build/asan/bin/kudu with pid 23103
I20260926 15:10:18.599365 12427 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 22743
I20260926 15:10:18.600502 22805 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:10:18.707309 12427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8uoJGv/build/asan/bin/kudu with pid 22743
2026-09-26T15:10:18Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4204 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435318278681-12427-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 15:10:18 dist-test-slave-865v krb5kdc[23281](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 15:10:18 dist-test-slave-865v krb5kdc[23281](info): set up 2 sockets
Sep 26 15:10:18 dist-test-slave-865v krb5kdc[23281](info): commencing operation
krb5kdc: starting...
W20260926 15:10:19.818570 12427 mini_kdc.cc:121] Time spent starting KDC: real 1.069s user 0.001s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435318278681-12427-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435318278681-12427-0/krb5kdc/test-user.keytab.
Sep 26 15:10:19 dist-test-slave-865v krb5kdc[23281](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435419, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T15:10:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T15:10:19Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.12.34.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.12.34.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.12.34.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435318278681-12427-0/krb5kdc/rangeradmin_127.12.34.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.12.34.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435318278681-12427-0/krb5kdc/rangeradmin_127.12.34.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.12.34.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.12.34.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.12.34.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435318278681-12427-0/krb5kdc/rangerlookup_127.12.34.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.12.34.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435318278681-12427-0/krb5kdc/rangerlookup_127.12.34.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.12.34.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.34.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.34.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435318278681-12427-0/krb5kdc/HTTP_127.12.34.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.12.34.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435318278681-12427-0/krb5kdc/HTTP_127.12.34.212@KRBTEST.COM.keytab.
I20260926 15:10:20.157373 12427 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435318278681-12427-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task8uoJGv/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435318278681-12427-0/minicluster-data/postgres -l logfile start
2026-09-26 15:10:22.730 UTC [23311] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-26 15:10:22.730 UTC [23311] LOG: listening on IPv4 address "127.12.34.212", port 38947
2026-09-26 15:10:22.734 UTC [23311] LOG: listening on Unix socket "/tmp/.s.PGSQL.38947"
2026-09-26 15:10:22.756 UTC [23314] LOG: database system was shut down at 2026-09-26 15:10:21 UTC
2026-09-26 15:10:22.759 UTC [23311] LOG: database system is ready to accept connections
I20260926 15:10:23.737751 12427 mini_postgres.cc:96] Postgres bound to 38947
2026-09-26 15:10:23.755 UTC [23322] FATAL: database "slave" does not exist
127.12.34.212:38947 - accepting connections
I20260926 15:10:23.756175 12427 mini_ranger.cc:161] Starting Ranger...
I20260926 15:10:23.782037 12427 mini_ranger.cc:84] Created miniranger Postgres user
I20260926 15:10:24.302690 12427 mini_ranger.cc:87] Created ranger Postgres database
I20260926 15:10:24.302835 12427 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435318278681-12427-0/minicluster-data/ranger-admin
2026-09-26 15:10:24,578 [I] DB FLAVOR :POSTGRES
2026-09-26 15:10:24,579 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 15:10:24,579 [I] Checking connection..
2026-09-26 15:10:24,579 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8uoJGv/build/asan/bin/postgresql.jar:/tmp/dist-test-task8uoJGv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.34.212:38947/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-26 15:10:24,817 [I] Checking connection passed.
2026-09-26 15:10:24,818 [I] --------- Verifying version history table ---------
2026-09-26 15:10:24,818 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8uoJGv/build/asan/bin/postgresql.jar:/tmp/dist-test-task8uoJGv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.34.212:38947/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 15:10:25,056 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 15:10:25,057 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8uoJGv/build/asan/bin/postgresql.jar:/tmp/dist-test-task8uoJGv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.34.212:38947/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 15:10:25,317 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 15:10:25,318 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-26 15:10:25,318 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8uoJGv/build/asan/bin/postgresql.jar:/tmp/dist-test-task8uoJGv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.34.212:38947/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task8uoJGv/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-26 15:10:25.588 UTC [23386] WARNING: there is no transaction in progress
2026-09-26 15:10:25,594 [I] create_dbversion_catalog.sql file imported successfully
2026-09-26 15:10:25,595 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8uoJGv/build/asan/bin/postgresql.jar:/tmp/dist-test-task8uoJGv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.34.212:38947/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 15:10:25,851 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-26 15:10:25,851 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 15:10:25,852 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8uoJGv/build/asan/bin/postgresql.jar:/tmp/dist-test-task8uoJGv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.34.212:38947/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-26 15:10:26,108 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8uoJGv/build/asan/bin/postgresql.jar:/tmp/dist-test-task8uoJGv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.34.212:38947/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-26 15:10:26,336 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8uoJGv/build/asan/bin/postgresql.jar:/tmp/dist-test-task8uoJGv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.34.212:38947/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-865v.c.gcp-upstream.internal','N') ;"
2026-09-26 15:10:26,588 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8uoJGv/build/asan/bin/postgresql.jar:/tmp/dist-test-task8uoJGv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.34.212:38947/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 15:10:26,817 [I] Table x_portal_user does not exist in database ranger
2026-09-26 15:10:26,818 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8uoJGv/build/asan/bin/postgresql.jar:/tmp/dist-test-task8uoJGv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.34.212:38947/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 15:10:27,072 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-26 15:10:27,072 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8uoJGv/build/asan/bin/postgresql.jar:/tmp/dist-test-task8uoJGv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.34.212:38947/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 15:10:27,297 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8uoJGv/build/asan/bin/postgresql.jar:/tmp/dist-test-task8uoJGv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.34.212:38947/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 15:10:27,548 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-26 15:10:27,548 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8uoJGv/build/asan/bin/postgresql.jar:/tmp/dist-test-task8uoJGv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.34.212:38947/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task8uoJGv/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-26 15:10:28.481 UTC [23494] WARNING: there is no transaction in progress
2026-09-26 15:10:28.497 UTC [23494] WARNING: there is no transaction in progress
2026-09-26 15:10:28.514 UTC [23494] WARNING: there is no transaction in progress
2026-09-26 15:10:28.528 UTC [23494] WARNING: there is no transaction in progress
2026-09-26 15:10:28.543 UTC [23494] WARNING: there is no transaction in progress
2026-09-26 15:10:28.562 UTC [23494] WARNING: there is no transaction in progress
2026-09-26 15:10:28.628 UTC [23494] WARNING: there is no transaction in progress
2026-09-26 15:10:28.635 UTC [23494] WARNING: there is no transaction in progress
2026-09-26 15:10:28.649 UTC [23494] WARNING: there is no transaction in progress
2026-09-26 15:10:28.659 UTC [23494] WARNING: there is no transaction in progress
2026-09-26 15:10:28.670 UTC [23494] WARNING: there is no transaction in progress
2026-09-26 15:10:28.682 UTC [23494] WARNING: there is no transaction in progress
2026-09-26 15:10:28.692 UTC [23494] WARNING: there is no transaction in progress
2026-09-26 15:10:28.698 UTC [23494] WARNING: there is no transaction in progress
2026-09-26 15:10:28.704 UTC [23494] WARNING: there is no transaction in progress
2026-09-26 15:10:28.710 UTC [23494] WARNING: there is no transaction in progress
2026-09-26 15:10:29,376 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-26 15:10:29,377 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8uoJGv/build/asan/bin/postgresql.jar:/tmp/dist-test-task8uoJGv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.34.212:38947/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-26 15:10:29,613 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-26 15:10:29,613 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8uoJGv/build/asan/bin/postgresql.jar:/tmp/dist-test-task8uoJGv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.34.212:38947/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 15:10:29,858 [I] Table x_portal_user already exists in database 'ranger'
2026-09-26 15:10:29,858 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8uoJGv/build/asan/bin/postgresql.jar:/tmp/dist-test-task8uoJGv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.34.212:38947/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 15:10:30,109 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-26 15:10:30,110 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8uoJGv/build/asan/bin/postgresql.jar:/tmp/dist-test-task8uoJGv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.34.212:38947/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 15:10:30,376 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8uoJGv/build/asan/bin/postgresql.jar:/tmp/dist-test-task8uoJGv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.34.212:38947/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 15:10:30,629 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8uoJGv/build/asan/bin/postgresql.jar:/tmp/dist-test-task8uoJGv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.34.212:38947/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-865v.c.gcp-upstream.internal';"
2026-09-26 15:10:30,874 [I] CORE_DB_SCHEMA import status has been updated
2026-09-26 15:10:30,874 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8uoJGv/build/asan/bin/postgresql.jar:/tmp/dist-test-task8uoJGv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.34.212:38947/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-26 15:10:31,108 [I] DB_PATCHES have already been applied
I20260926 15:10:31.115208 12427 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435318278681-12427-0/minicluster-data/ranger-admin:/tmp/dist-test-task8uoJGv/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task8uoJGv/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task8uoJGv/build/asan/bin/postgresql.jar:/tmp/dist-test-task8uoJGv/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260926 15:10:31.115319 12427 mini_ranger.cc:193] Using host: 127.12.34.212
I20260926 15:10:31.130299 12427 mini_ranger.cc:227] Ranger admin URL: http://127.12.34.212:38009
Sep 26, 2026 3:10:31 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 3:10:31 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 3:10:31 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435318278681-12427-0/minicluster-data/ranger-admin/ews/webapp
Sep 26, 2026 3:10:31 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435318278681-12427-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 26, 2026 3:10:31 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435318278681-12427-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 26, 2026 3:10:32 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435318278681-12427-0/minicluster-data/ranger-admin/ews/webapp].
Sep 26, 2026 3:10:32 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260926 15:10:32.268028 12427 mini_ranger.cc:160] Time spent starting Ranger: real 8.512s user 0.000s sys 0.037s
Sep 26, 2026 3:10:52 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 26, 2026 3:10:52 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 26, 2026 3:10:52 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 26, 2026 3:10:52 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 26, 2026 3:10:52 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 26, 2026 3:10:52 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 26, 2026 3:10:52 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 26, 2026 3:10:52 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 26, 2026 3:10:52 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 26, 2026 3:10:52 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 26, 2026 3:10:52 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 26, 2026 3:10:52 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 26, 2026 3:10:52 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 26, 2026 3:10:52 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 26, 2026 3:10:52 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 26, 2026 3:10:52 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 26, 2026 3:10:52 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 26, 2026 3:10:53 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260926 15:10:53.807369 12427 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.12.34.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.12.34.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.12.34.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435318278681-12427-0/krb5kdc/rangerkms_127.12.34.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.12.34.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435318278681-12427-0/krb5kdc/rangerkms_127.12.34.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.12.34.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.12.34.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.34.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435318278681-12427-0/krb5kdc/HTTP_127.12.34.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.12.34.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435318278681-12427-0/krb5kdc/HTTP_127.12.34.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 26 15:10:53 dist-test-slave-865v krb5kdc[23281](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435453, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260926 15:10:53.976460 12427 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260926 15:10:53.995738 12427 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260926 15:10:54.910998 12427 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260926 15:10:54.911159 12427 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435318278681-12427-0/minicluster-data/ranger-kms
I20260926 15:10:54.911304 12427 mini_ranger_kms.cc:227] Using postgres at 127.12.34.212:38947
2026-09-26 15:10:55,321 [I] DB FLAVOR :POSTGRES
2026-09-26 15:10:55,321 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 15:10:55,322 [I] Checking connection
2026-09-26 15:10:55,322 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8uoJGv/build/asan/bin/postgresql.jar:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435318278681-12427-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.34.212:38947/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-26 15:10:55,544 [I] connection success
2026-09-26 15:10:55,545 [I] --------- Verifying Ranger DB tables ---------
2026-09-26 15:10:55,545 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-26 15:10:55,545 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8uoJGv/build/asan/bin/postgresql.jar:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435318278681-12427-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.34.212:38947/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-26 15:10:55,774 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-26 15:10:55,774 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 15:10:55,774 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-26 15:10:55,775 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8uoJGv/build/asan/bin/postgresql.jar:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435318278681-12427-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.34.212:38947/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435318278681-12427-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-26 15:10:56,066 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260926 15:10:56.530088 12427 mini_ranger_kms.cc:310] Created kms service
I20260926 15:10:56.731238 12427 mini_ranger_kms.cc:326] Created kudu user
I20260926 15:10:56.775934 12427 mini_ranger_kms.cc:343] Created rangerkms user
Sep 26 15:10:56 dist-test-slave-865v krb5kdc[23281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435453, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.12.34.212@KRBTEST.COM
I20260926 15:10:57.178131 12427 mini_ranger_kms.cc:384] Added ranger policy
I20260926 15:10:57.178457 12427 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435318278681-12427-0/minicluster-data/ranger-kms:/tmp/dist-test-task8uoJGv/build/asan/bin/postgresql.jar:/tmp/dist-test-task8uoJGv/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task8uoJGv/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260926 15:10:57.178524 12427 mini_ranger_kms.cc:240] Using host: 127.12.34.212
I20260926 15:10:57.199101 12427 mini_ranger_kms.cc:276] Ranger KMS PID: 23674
I20260926 15:10:57.199297 12427 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.12.34.212:52369
Sep 26, 2026 3:10:58 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 3:10:58 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 3:10:58 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435318278681-12427-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 26, 2026 3:10:58 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435318278681-12427-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 26, 2026 3:10:58 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435318278681-12427-0/minicluster-data/ranger-kms/ews/webapp].
Sep 26, 2026 3:10:58 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 26 15:10:59 dist-test-slave-865v krb5kdc[23281](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790435459, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.12.34.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:10:59.482224 12427 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.506s user 0.126s sys 0.298s
I20260926 15:10:59.482499 12427 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260926 15:10:59.482735 12427 mini_ranger_kms.cc:401] 127.12.34.212:52369/kms/v1/keys
Sep 26 15:10:59 dist-test-slave-865v krb5kdc[23281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435459, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.12.34.212@KRBTEST.COM for HTTP/127.12.34.212@KRBTEST.COM
Sep 26 15:11:00 dist-test-slave-865v krb5kdc[23281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435459, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.12.34.212@KRBTEST.COM for HTTP/127.12.34.212@KRBTEST.COM
Sep 26 15:11:02 dist-test-slave-865v krb5kdc[23281](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435462, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.12.34.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.34.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.34.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.254.keytab.
Entry for principal kudu/127.12.34.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.254.keytab.
I20260926 15:11:02.552606 12427 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8uoJGv/build/asan/bin/kudu
/tmp/dist-test-task8uoJGv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435318278681-12427-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435318278681-12427-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435318278681-12427-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.34.254:40719
--webserver_interface=127.12.34.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.254.keytab
--principal=kudu/127.12.34.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.34.212:35787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.34.254:40719
--ranger_config_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435318278681-12427-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.34.212:52369/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435318278681-12427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435318278681-12427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435318278681-12427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:11:02.897969 23725 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:11:02.898233 23725 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:11:02.898373 23725 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:11:02.905511 23725 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:11:02.905606 23725 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:11:02.905690 23725 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:11:02.905786 23725 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:11:02.905874 23725 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:11:02.915887 23725 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.34.212:35787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435318278681-12427-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435318278681-12427-0/minicluster-data/master-0/wal
--ranger_kms_url=127.12.34.212:52369/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.12.34.254:40719
--ranger_config_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435318278681-12427-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.254.keytab
--principal=kudu/127.12.34.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.34.254:40719
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.34.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435318278681-12427-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:11:02.917132 23725 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:11:02.918411 23725 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:11:02.930135 23730 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:11:02.930197 23731 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:11:02.930220 23733 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:11:02.930977 23725 server_base.cc:1034] running on GCE node
I20260926 15:11:02.931663 23725 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:11:02.932898 23725 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:11:02.934154 23725 hybrid_clock.cc:648] HybridClock initialized: now 1790435462934111 us; error 28 us; skew 500 ppm
I20260926 15:11:02.934525 23725 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:11:02 dist-test-slave-865v krb5kdc[23281](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435462, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:11:02.937443 23725 init.cc:377] Logged in from keytab as kudu/127.12.34.254@KRBTEST.COM (short username kudu)
I20260926 15:11:02.938738 23725 webserver.cc:460] Webserver started at http://127.12.34.254:40023/ using document root <none> and password file <none>
I20260926 15:11:02.939333 23725 fs_manager.cc:359] Metadata directory not provided
I20260926 15:11:02.939462 23725 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:11:02.939751 23725 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 15:11:02 dist-test-slave-865v krb5kdc[23281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435462, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.254@KRBTEST.COM for HTTP/127.12.34.212@KRBTEST.COM
I20260926 15:11:02.995074 23725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435318278681-12427-0/minicluster-data/master-0/data/instance:
uuid: "dee3b6427b5e4aab8a67763313094ee5"
format_stamp: "Formatted at 2026-09-26 15:11:02 on dist-test-slave-865v"
server_key: "639b62082f1d57ab5e792f0356dfc603"
server_key_iv: "7ef405393548fa18c301d37e9c34df58"
server_key_version: "kuduclusterkey@0"
I20260926 15:11:02.995911 23725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435318278681-12427-0/minicluster-data/master-0/wal/instance:
uuid: "dee3b6427b5e4aab8a67763313094ee5"
format_stamp: "Formatted at 2026-09-26 15:11:02 on dist-test-slave-865v"
server_key: "639b62082f1d57ab5e792f0356dfc603"
server_key_iv: "7ef405393548fa18c301d37e9c34df58"
server_key_version: "kuduclusterkey@0"
I20260926 15:11:03.002054 23725 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260926 15:11:03.021960 23740 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:11:03.023650 23725 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 15:11:03.023829 23725 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435318278681-12427-0/minicluster-data/master-0/data,/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435318278681-12427-0/minicluster-data/master-0/wal
uuid: "dee3b6427b5e4aab8a67763313094ee5"
format_stamp: "Formatted at 2026-09-26 15:11:02 on dist-test-slave-865v"
server_key: "639b62082f1d57ab5e792f0356dfc603"
server_key_iv: "7ef405393548fa18c301d37e9c34df58"
server_key_version: "kuduclusterkey@0"
I20260926 15:11:03.024070 23725 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435318278681-12427-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435318278681-12427-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435318278681-12427-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:11:03.046357 23725 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:11:03.051188 23725 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:11:03.051502 23725 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:11:03.071933 23725 rpc_server.cc:307] RPC server started. Bound to: 127.12.34.254:40719
I20260926 15:11:03.071933 23791 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.34.254:40719 every 8 connection(s)
I20260926 15:11:03.073607 23725 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
I20260926 15:11:03.076653 12427 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8uoJGv/build/asan/bin/kudu as pid 23725
I20260926 15:11:03.076838 12427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435318278681-12427-0/minicluster-data/master-0/wal/instance
I20260926 15:11:03.077420 23792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:11:03.077669 12427 external_mini_cluster.cc:1428] Setting key 49b1482205377d81745305297cf5ec29
I20260926 15:11:03.095438 23792 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dee3b6427b5e4aab8a67763313094ee5: Bootstrap starting.
I20260926 15:11:03.099639 23792 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dee3b6427b5e4aab8a67763313094ee5: Neither blocks nor log segments found. Creating new log.
I20260926 15:11:03.100687 23792 log.cc:824] T 00000000000000000000000000000000 P dee3b6427b5e4aab8a67763313094ee5: Log is configured to *not* fsync() on all Append() calls
I20260926 15:11:03.104025 23792 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dee3b6427b5e4aab8a67763313094ee5: No bootstrap required, opened a new log
Sep 26 15:11:03 dist-test-slave-865v krb5kdc[23281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435462, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.34.254@KRBTEST.COM
I20260926 15:11:03.114614 23792 raft_consensus.cc:348] T 00000000000000000000000000000000 P dee3b6427b5e4aab8a67763313094ee5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dee3b6427b5e4aab8a67763313094ee5" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 40719 } }
I20260926 15:11:03.115000 23792 raft_consensus.cc:374] T 00000000000000000000000000000000 P dee3b6427b5e4aab8a67763313094ee5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:11:03.115103 23792 raft_consensus.cc:729] T 00000000000000000000000000000000 P dee3b6427b5e4aab8a67763313094ee5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dee3b6427b5e4aab8a67763313094ee5, State: Initialized, Role: FOLLOWER
I20260926 15:11:03.115782 23792 consensus_queue.cc:260] T 00000000000000000000000000000000 P dee3b6427b5e4aab8a67763313094ee5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dee3b6427b5e4aab8a67763313094ee5" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 40719 } }
I20260926 15:11:03.116036 23792 raft_consensus.cc:388] T 00000000000000000000000000000000 P dee3b6427b5e4aab8a67763313094ee5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 15:11:03.116187 23792 raft_consensus.cc:482] T 00000000000000000000000000000000 P dee3b6427b5e4aab8a67763313094ee5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 15:11:03.116370 23792 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dee3b6427b5e4aab8a67763313094ee5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:11:03.118899 23795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:11:03.079655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42862 (local address 127.12.34.254:40719)
0926 15:11:03.094636 (+ 14981us) server_negotiation.cc:206] Beginning negotiation
0926 15:11:03.094652 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 15:11:03.094702 (+ 50us) server_negotiation.cc:407] Connection header received
0926 15:11:03.097017 (+ 2315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:11:03.097046 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:11:03.097998 (+ 952us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:11:03.098443 (+ 445us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:11:03.099910 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:11:03.102028 (+ 2118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:11:03.103806 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:11:03.104766 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:11:03.104947 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:11:03.109704 (+ 4757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:11:03.109720 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:11:03.109739 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:11:03.109788 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:11:03.114189 (+ 4401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:11:03.115024 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:11:03.115033 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:11:03.115042 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:11:03.115231 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:11:03.115648 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:11:03.115653 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:11:03.115655 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:11:03.116340 (+ 685us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:11:03.116595 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0926 15:11:03.117131 (+ 536us) server_negotiation.cc:299] Negotiation successful
0926 15:11:03.117576 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14748,"thread_start_us":14554,"threads_started":1}
I20260926 15:11:03.119230 23792 raft_consensus.cc:504] T 00000000000000000000000000000000 P dee3b6427b5e4aab8a67763313094ee5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dee3b6427b5e4aab8a67763313094ee5" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 40719 } }
I20260926 15:11:03.119784 23792 leader_election.cc:302] T 00000000000000000000000000000000 P dee3b6427b5e4aab8a67763313094ee5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dee3b6427b5e4aab8a67763313094ee5; no voters:
I20260926 15:11:03.120574 23792 leader_election.cc:288] T 00000000000000000000000000000000 P dee3b6427b5e4aab8a67763313094ee5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 15:11:03.120663 23797 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dee3b6427b5e4aab8a67763313094ee5 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:11:03.122953 23797 raft_consensus.cc:686] T 00000000000000000000000000000000 P dee3b6427b5e4aab8a67763313094ee5 [term 1 LEADER]: Becoming Leader. State: Replica: dee3b6427b5e4aab8a67763313094ee5, State: Running, Role: LEADER
I20260926 15:11:03.123555 23797 consensus_queue.cc:237] T 00000000000000000000000000000000 P dee3b6427b5e4aab8a67763313094ee5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dee3b6427b5e4aab8a67763313094ee5" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 40719 } }
I20260926 15:11:03.125612 23797 sys_catalog.cc:455] T 00000000000000000000000000000000 P dee3b6427b5e4aab8a67763313094ee5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dee3b6427b5e4aab8a67763313094ee5. Latest consensus state: current_term: 1 leader_uuid: "dee3b6427b5e4aab8a67763313094ee5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dee3b6427b5e4aab8a67763313094ee5" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 40719 } } }
I20260926 15:11:03.129135 23797 sys_catalog.cc:458] T 00000000000000000000000000000000 P dee3b6427b5e4aab8a67763313094ee5 [sys.catalog]: This master's current role is: LEADER
I20260926 15:11:03.129297 23799 sys_catalog.cc:455] T 00000000000000000000000000000000 P dee3b6427b5e4aab8a67763313094ee5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dee3b6427b5e4aab8a67763313094ee5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dee3b6427b5e4aab8a67763313094ee5" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 40719 } } }
I20260926 15:11:03.129770 23792 sys_catalog.cc:564] T 00000000000000000000000000000000 P dee3b6427b5e4aab8a67763313094ee5 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:11:03.130617 23802 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:11:03.136113 23799 sys_catalog.cc:458] T 00000000000000000000000000000000 P dee3b6427b5e4aab8a67763313094ee5 [sys.catalog]: This master's current role is: LEADER
I20260926 15:11:03.141233 23802 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 15:11:03.158561 23802 catalog_manager.cc:1347] Generated new cluster ID: ab922b25438c4e7db60310c080c0db91
I20260926 15:11:03.158702 23802 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 15:11:03.184522 23792 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435318278681-12427-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260926 15:11:03.185671 23802 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 15:11:03.188019 23802 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:11:03.206805 23802 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dee3b6427b5e4aab8a67763313094ee5: Generated new TSK 0
I20260926 15:11:03.208951 23802 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.34.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.34.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.34.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.193.keytab.
Entry for principal kudu/127.12.34.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.193.keytab.
I20260926 15:11:04.017046 12427 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8uoJGv/build/asan/bin/kudu
/tmp/dist-test-task8uoJGv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435318278681-12427-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435318278681-12427-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435318278681-12427-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435318278681-12427-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.34.212:52369/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.34.193:0
--local_ip_for_outbound_sockets=127.12.34.193
--webserver_interface=127.12.34.193
--webserver_port=0
--tserver_master_addrs=127.12.34.254:40719
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.193.keytab
--principal=kudu/127.12.34.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.34.212:35787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435318278681-12427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435318278681-12427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435318278681-12427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 15:11:04 dist-test-slave-865v krb5kdc[23281](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790435464, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260926 15:11:04.409276 23837 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:11:04.409713 23837 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:11:04.409888 23837 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:11:04.417306 23837 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:11:04.417447 23837 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:11:04.417644 23837 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.34.193
I20260926 15:11:04.427910 23837 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.34.212:35787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435318278681-12427-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435318278681-12427-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.12.34.212:52369/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.193.keytab
--principal=kudu/127.12.34.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.34.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435318278681-12427-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.34.193
--webserver_port=0
--tserver_master_addrs=127.12.34.254:40719
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.34.193
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435318278681-12427-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:11:04.429345 23837 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:11:04.430816 23837 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:11:04.453776 23847 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:11:04.453964 23848 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:11:04.461295 23850 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:11:04.461640 23837 server_base.cc:1034] running on GCE node
I20260926 15:11:04.462471 23837 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:11:04.463721 23837 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:11:04.464982 23837 hybrid_clock.cc:648] HybridClock initialized: now 1790435464464918 us; error 49 us; skew 500 ppm
I20260926 15:11:04.465402 23837 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:11:04 dist-test-slave-865v krb5kdc[23281](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435464, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:11:04.469497 23837 init.cc:377] Logged in from keytab as kudu/127.12.34.193@KRBTEST.COM (short username kudu)
I20260926 15:11:04.505652 23837 webserver.cc:460] Webserver started at http://127.12.34.193:39981/ using document root <none> and password file <none>
I20260926 15:11:04.506768 23837 fs_manager.cc:359] Metadata directory not provided
I20260926 15:11:04.506935 23837 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:11:04.507311 23837 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 15:11:04 dist-test-slave-865v krb5kdc[23281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435464, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.193@KRBTEST.COM for HTTP/127.12.34.212@KRBTEST.COM
I20260926 15:11:04.524140 23837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435318278681-12427-0/minicluster-data/ts-0/data/instance:
uuid: "85eed29572b54740a074baa22c71d446"
format_stamp: "Formatted at 2026-09-26 15:11:04 on dist-test-slave-865v"
server_key: "f9b821acb5c38cf27f079cfe061867fa"
server_key_iv: "315fb51705c4faf49297a5f74e8c0c94"
server_key_version: "kuduclusterkey@0"
I20260926 15:11:04.525318 23837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435318278681-12427-0/minicluster-data/ts-0/wal/instance:
uuid: "85eed29572b54740a074baa22c71d446"
format_stamp: "Formatted at 2026-09-26 15:11:04 on dist-test-slave-865v"
server_key: "f9b821acb5c38cf27f079cfe061867fa"
server_key_iv: "315fb51705c4faf49297a5f74e8c0c94"
server_key_version: "kuduclusterkey@0"
I20260926 15:11:04.549499 23837 fs_manager.cc:691] Time spent creating directory manager: real 0.024s user 0.006s sys 0.016s
I20260926 15:11:04.585947 23857 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:11:04.590580 23837 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.002s sys 0.000s
I20260926 15:11:04.590788 23837 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435318278681-12427-0/minicluster-data/ts-0/data,/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435318278681-12427-0/minicluster-data/ts-0/wal
uuid: "85eed29572b54740a074baa22c71d446"
format_stamp: "Formatted at 2026-09-26 15:11:04 on dist-test-slave-865v"
server_key: "f9b821acb5c38cf27f079cfe061867fa"
server_key_iv: "315fb51705c4faf49297a5f74e8c0c94"
server_key_version: "kuduclusterkey@0"
I20260926 15:11:04.591063 23837 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435318278681-12427-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435318278681-12427-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435318278681-12427-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:11:04.627607 23837 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:11:04.646754 23837 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:11:04.647233 23837 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:11:04.665544 23837 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:11:04.679795 23837 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:11:04.679975 23837 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260926 15:11:04.680166 23837 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:11:04.680266 23837 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 15:11:04 dist-test-slave-865v krb5kdc[23281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435464, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.254@KRBTEST.COM for HTTP/127.12.34.212@KRBTEST.COM
I20260926 15:11:04.909466 23837 rpc_server.cc:307] RPC server started. Bound to: 127.12.34.193:46797
I20260926 15:11:04.909533 23969 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.34.193:46797 every 8 connection(s)
I20260926 15:11:04.911314 23837 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435318278681-12427-0/minicluster-data/ts-0/data/info.pb
I20260926 15:11:04.921262 12427 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8uoJGv/build/asan/bin/kudu as pid 23837
I20260926 15:11:04.921480 12427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435318278681-12427-0/minicluster-data/ts-0/wal/instance
I20260926 15:11:04.922313 12427 external_mini_cluster.cc:1428] Setting key d3920b869fe9a6d8552db6d42c324dd0
Sep 26 15:11:04 dist-test-slave-865v krb5kdc[23281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435464, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.193@KRBTEST.COM for kudu/127.12.34.254@KRBTEST.COM
I20260926 15:11:04.942767 23972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:11:04.914395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.193:42403 (local address 127.12.34.254:40719)
0926 15:11:04.914880 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0926 15:11:04.914887 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:11:04.915840 (+ 953us) server_negotiation.cc:407] Connection header received
0926 15:11:04.919746 (+ 3906us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:11:04.919755 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:11:04.919864 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:11:04.919952 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:11:04.925099 (+ 5147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:11:04.926267 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:11:04.929718 (+ 3451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:11:04.930339 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:11:04.930437 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:11:04.935227 (+ 4790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:11:04.935233 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:11:04.935238 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:11:04.935283 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:11:04.939110 (+ 3827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:11:04.940057 (+ 947us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:11:04.940062 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:11:04.940066 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:11:04.940216 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:11:04.940699 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:11:04.940704 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:11:04.940719 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:11:04.941048 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:11:04.941216 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0926 15:11:04.942175 (+ 959us) server_negotiation.cc:299] Negotiation successful
0926 15:11:04.942418 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":1694,"threads_started":1}
I20260926 15:11:04.944571 23973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:11:04.914907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:40719 (local address 127.12.34.193:42403)
0926 15:11:04.915694 (+ 787us) negotiation.cc:107] Waiting for socket to connect
0926 15:11:04.915745 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 15:11:04.919370 (+ 3625us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:11:04.920519 (+ 1149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:11:04.920531 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:11:04.923803 (+ 3272us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:11:04.924858 (+ 1055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:11:04.924886 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:11:04.926477 (+ 1591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:11:04.926483 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:11:04.929554 (+ 3071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:11:04.929569 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:11:04.930826 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:11:04.930832 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:11:04.931088 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:11:04.931882 (+ 794us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:11:04.931911 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:11:04.934975 (+ 3064us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:11:04.939319 (+ 4344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:11:04.939328 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:11:04.939346 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:11:04.939900 (+ 554us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:11:04.940355 (+ 455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:11:04.940359 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:11:04.940362 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:11:04.940579 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:11:04.941299 (+ 720us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:11:04.941306 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:11:04.941733 (+ 427us) client_negotiation.cc:769] Sending connection context
0926 15:11:04.943082 (+ 1349us) client_negotiation.cc:240] Negotiation successful
0926 15:11:04.943506 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":312,"threads_started":1}
I20260926 15:11:04.946476 23970 heartbeater.cc:347] Connected to a master server at 127.12.34.254:40719
I20260926 15:11:04.946792 23970 heartbeater.cc:464] Registering TS with master...
I20260926 15:11:04.947592 23970 heartbeater.cc:511] Master 127.12.34.254:40719 requested a full tablet report, sending...
I20260926 15:11:04.950237 23757 ts_manager.cc:194] Registered new tserver with Master: 85eed29572b54740a074baa22c71d446 (127.12.34.193:46797)
I20260926 15:11:04.952363 23757 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.34.193@KRBTEST.COM'} at 127.12.34.193:42403
WARNING: no policy specified for kudu/127.12.34.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.34.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.34.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.194.keytab.
Entry for principal kudu/127.12.34.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.194.keytab.
I20260926 15:11:04.992929 12427 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8uoJGv/build/asan/bin/kudu
/tmp/dist-test-task8uoJGv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435318278681-12427-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435318278681-12427-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435318278681-12427-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435318278681-12427-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.34.212:52369/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.34.194:0
--local_ip_for_outbound_sockets=127.12.34.194
--webserver_interface=127.12.34.194
--webserver_port=0
--tserver_master_addrs=127.12.34.254:40719
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.194.keytab
--principal=kudu/127.12.34.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.34.212:35787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435318278681-12427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435318278681-12427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435318278681-12427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 15:11:05 dist-test-slave-865v krb5kdc[23281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435464, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.254@KRBTEST.COM for HTTP/127.12.34.212@KRBTEST.COM
W20260926 15:11:05.489005 23976 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:11:05.489578 23976 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:11:05.489809 23976 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:11:05.503017 23976 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:11:05.503388 23976 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:11:05.503687 23976 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.34.194
I20260926 15:11:05.519368 23976 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.34.212:35787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435318278681-12427-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435318278681-12427-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.12.34.212:52369/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.194.keytab
--principal=kudu/127.12.34.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.34.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435318278681-12427-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.34.194
--webserver_port=0
--tserver_master_addrs=127.12.34.254:40719
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.34.194
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435318278681-12427-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:11:05.521247 23976 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:11:05.522979 23976 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:11:05.561064 23983 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:11:05.565426 23984 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:11:05.573982 23976 server_base.cc:1034] running on GCE node
W20260926 15:11:05.575915 23987 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:11:05.576884 23976 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:11:05.583084 23976 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:11:05.584374 23976 hybrid_clock.cc:648] HybridClock initialized: now 1790435465584271 us; error 104 us; skew 500 ppm
I20260926 15:11:05.584733 23976 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:11:05 dist-test-slave-865v krb5kdc[23281](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435465, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:11:05.590047 23976 init.cc:377] Logged in from keytab as kudu/127.12.34.194@KRBTEST.COM (short username kudu)
I20260926 15:11:05.600989 23976 webserver.cc:460] Webserver started at http://127.12.34.194:41435/ using document root <none> and password file <none>
I20260926 15:11:05.601842 23976 fs_manager.cc:359] Metadata directory not provided
I20260926 15:11:05.601999 23976 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:11:05.602347 23976 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 15:11:05 dist-test-slave-865v krb5kdc[23281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435465, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.194@KRBTEST.COM for HTTP/127.12.34.212@KRBTEST.COM
I20260926 15:11:05.620422 23976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435318278681-12427-0/minicluster-data/ts-1/data/instance:
uuid: "ae3c0a0842794b77ba9995dff32974a9"
format_stamp: "Formatted at 2026-09-26 15:11:05 on dist-test-slave-865v"
server_key: "67e451ce3fcfe8b7ec33b8952a20ad62"
server_key_iv: "41421319c79261712f5a597ef7cc165b"
server_key_version: "kuduclusterkey@0"
I20260926 15:11:05.621562 23976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435318278681-12427-0/minicluster-data/ts-1/wal/instance:
uuid: "ae3c0a0842794b77ba9995dff32974a9"
format_stamp: "Formatted at 2026-09-26 15:11:05 on dist-test-slave-865v"
server_key: "67e451ce3fcfe8b7ec33b8952a20ad62"
server_key_iv: "41421319c79261712f5a597ef7cc165b"
server_key_version: "kuduclusterkey@0"
I20260926 15:11:05.639050 23976 fs_manager.cc:691] Time spent creating directory manager: real 0.017s user 0.008s sys 0.008s
I20260926 15:11:05.656492 24000 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:11:05.658008 23976 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 15:11:05.658181 23976 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435318278681-12427-0/minicluster-data/ts-1/data,/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435318278681-12427-0/minicluster-data/ts-1/wal
uuid: "ae3c0a0842794b77ba9995dff32974a9"
format_stamp: "Formatted at 2026-09-26 15:11:05 on dist-test-slave-865v"
server_key: "67e451ce3fcfe8b7ec33b8952a20ad62"
server_key_iv: "41421319c79261712f5a597ef7cc165b"
server_key_version: "kuduclusterkey@0"
I20260926 15:11:05.658412 23976 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435318278681-12427-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435318278681-12427-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435318278681-12427-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:11:05.671984 23976 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:11:05.676354 23976 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:11:05.676664 23976 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:11:05.677734 23976 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:11:05.679170 23976 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:11:05.679239 23976 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:11:05.679484 23976 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:11:05.679580 23976 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:11:05.718974 23976 rpc_server.cc:307] RPC server started. Bound to: 127.12.34.194:45599
I20260926 15:11:05.718993 24112 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.34.194:45599 every 8 connection(s)
I20260926 15:11:05.720386 23976 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435318278681-12427-0/minicluster-data/ts-1/data/info.pb
I20260926 15:11:05.723527 12427 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8uoJGv/build/asan/bin/kudu as pid 23976
I20260926 15:11:05.723683 12427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435318278681-12427-0/minicluster-data/ts-1/wal/instance
I20260926 15:11:05.724182 12427 external_mini_cluster.cc:1428] Setting key 4dce7be415e5c29dc61992bf000a8748
Sep 26 15:11:05 dist-test-slave-865v krb5kdc[23281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435465, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.194@KRBTEST.COM for kudu/127.12.34.254@KRBTEST.COM
I20260926 15:11:05.751062 24115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:11:05.723645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.194:45965 (local address 127.12.34.254:40719)
0926 15:11:05.733493 (+ 9848us) server_negotiation.cc:206] Beginning negotiation
0926 15:11:05.733498 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:11:05.733515 (+ 17us) server_negotiation.cc:407] Connection header received
0926 15:11:05.735378 (+ 1863us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:11:05.735386 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:11:05.735492 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:11:05.735635 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:11:05.737484 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:11:05.738643 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:11:05.740237 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:11:05.740636 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:11:05.740766 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:11:05.744480 (+ 3714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:11:05.744486 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:11:05.744490 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:11:05.744521 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:11:05.747730 (+ 3209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:11:05.748586 (+ 856us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:11:05.748590 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:11:05.748593 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:11:05.748707 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:11:05.749205 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:11:05.749219 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:11:05.749222 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:11:05.749573 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:11:05.749752 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0926 15:11:05.750463 (+ 711us) server_negotiation.cc:299] Negotiation successful
0926 15:11:05.750686 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9733,"thread_start_us":9616,"threads_started":1}
I20260926 15:11:05.752195 24116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:11:05.724358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:40719 (local address 127.12.34.194:45965)
0926 15:11:05.725453 (+ 1095us) negotiation.cc:107] Waiting for socket to connect
0926 15:11:05.725518 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0926 15:11:05.734735 (+ 9217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:11:05.736122 (+ 1387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:11:05.736132 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:11:05.736617 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:11:05.737318 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:11:05.737337 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:11:05.738819 (+ 1482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:11:05.738829 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:11:05.740079 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:11:05.740092 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:11:05.740878 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:11:05.740885 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:11:05.741184 (+ 299us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:11:05.742145 (+ 961us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:11:05.742186 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:11:05.744243 (+ 2057us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:11:05.747897 (+ 3654us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:11:05.747907 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:11:05.747926 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:11:05.748387 (+ 461us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:11:05.748853 (+ 466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:11:05.748858 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:11:05.748862 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:11:05.749057 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:11:05.749730 (+ 673us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:11:05.749738 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:11:05.750083 (+ 345us) client_negotiation.cc:769] Sending connection context
0926 15:11:05.750353 (+ 270us) client_negotiation.cc:240] Negotiation successful
0926 15:11:05.750909 (+ 556us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":800,"thread_start_us":452,"threads_started":1}
WARNING: no policy specified for kudu/127.12.34.195@KRBTEST.COM; defaulting to no policy
I20260926 15:11:05.753834 24113 heartbeater.cc:347] Connected to a master server at 127.12.34.254:40719
I20260926 15:11:05.754145 24113 heartbeater.cc:464] Registering TS with master...
I20260926 15:11:05.754914 24113 heartbeater.cc:511] Master 127.12.34.254:40719 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.34.195@KRBTEST.COM" created.
I20260926 15:11:05.756515 23757 ts_manager.cc:194] Registered new tserver with Master: ae3c0a0842794b77ba9995dff32974a9 (127.12.34.194:45599)
I20260926 15:11:05.757908 23757 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.34.194@KRBTEST.COM'} at 127.12.34.194:45965
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.34.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.195.keytab.
Entry for principal kudu/127.12.34.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.195.keytab.
I20260926 15:11:05.794293 12427 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8uoJGv/build/asan/bin/kudu
/tmp/dist-test-task8uoJGv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435318278681-12427-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435318278681-12427-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435318278681-12427-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435318278681-12427-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.34.212:52369/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.34.195:0
--local_ip_for_outbound_sockets=127.12.34.195
--webserver_interface=127.12.34.195
--webserver_port=0
--tserver_master_addrs=127.12.34.254:40719
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.195.keytab
--principal=kudu/127.12.34.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.34.212:35787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435318278681-12427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435318278681-12427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435318278681-12427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 15:11:05.956283 23970 heartbeater.cc:503] Master 127.12.34.254:40719 was elected leader, sending a full tablet report...
W20260926 15:11:06.161422 24119 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:11:06.161717 24119 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:11:06.161913 24119 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:11:06.169070 24119 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:11:06.169286 24119 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:11:06.169454 24119 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.34.195
I20260926 15:11:06.179253 24119 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.34.212:35787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435318278681-12427-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435318278681-12427-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.12.34.212:52369/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.195.keytab
--principal=kudu/127.12.34.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.34.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435318278681-12427-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.34.195
--webserver_port=0
--tserver_master_addrs=127.12.34.254:40719
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.34.195
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435318278681-12427-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:11:06.180578 24119 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:11:06.181991 24119 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:11:06.196096 24124 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:11:06.196074 24125 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:11:06.196624 24127 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:11:06.198333 24119 server_base.cc:1034] running on GCE node
I20260926 15:11:06.198922 24119 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:11:06.199857 24119 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:11:06.201051 24119 hybrid_clock.cc:648] HybridClock initialized: now 1790435466201001 us; error 52 us; skew 500 ppm
I20260926 15:11:06.201436 24119 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:11:06 dist-test-slave-865v krb5kdc[23281](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435466, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:11:06.204649 24119 init.cc:377] Logged in from keytab as kudu/127.12.34.195@KRBTEST.COM (short username kudu)
I20260926 15:11:06.206079 24119 webserver.cc:460] Webserver started at http://127.12.34.195:34261/ using document root <none> and password file <none>
I20260926 15:11:06.206753 24119 fs_manager.cc:359] Metadata directory not provided
I20260926 15:11:06.206880 24119 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:11:06.207208 24119 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 15:11:06 dist-test-slave-865v krb5kdc[23281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435466, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.195@KRBTEST.COM for HTTP/127.12.34.212@KRBTEST.COM
I20260926 15:11:06.220209 24119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435318278681-12427-0/minicluster-data/ts-2/data/instance:
uuid: "3b9d7eb299a046398180424ee41b4e77"
format_stamp: "Formatted at 2026-09-26 15:11:06 on dist-test-slave-865v"
server_key: "3669d07497a027d813899926f8eca2ba"
server_key_iv: "c0093df7cd24e0bffbf3b5d979d29c6b"
server_key_version: "kuduclusterkey@0"
I20260926 15:11:06.220999 24119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435318278681-12427-0/minicluster-data/ts-2/wal/instance:
uuid: "3b9d7eb299a046398180424ee41b4e77"
format_stamp: "Formatted at 2026-09-26 15:11:06 on dist-test-slave-865v"
server_key: "3669d07497a027d813899926f8eca2ba"
server_key_iv: "c0093df7cd24e0bffbf3b5d979d29c6b"
server_key_version: "kuduclusterkey@0"
I20260926 15:11:06.227119 24119 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 15:11:06.241595 24134 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:11:06.243266 24119 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 15:11:06.243439 24119 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435318278681-12427-0/minicluster-data/ts-2/data,/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435318278681-12427-0/minicluster-data/ts-2/wal
uuid: "3b9d7eb299a046398180424ee41b4e77"
format_stamp: "Formatted at 2026-09-26 15:11:06 on dist-test-slave-865v"
server_key: "3669d07497a027d813899926f8eca2ba"
server_key_iv: "c0093df7cd24e0bffbf3b5d979d29c6b"
server_key_version: "kuduclusterkey@0"
I20260926 15:11:06.243646 24119 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435318278681-12427-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435318278681-12427-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435318278681-12427-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:11:06.264912 24119 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:11:06.269269 24119 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:11:06.269563 24119 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:11:06.270486 24119 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:11:06.271785 24119 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:11:06.271853 24119 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:11:06.271972 24119 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:11:06.272020 24119 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:11:06.306711 24119 rpc_server.cc:307] RPC server started. Bound to: 127.12.34.195:37423
I20260926 15:11:06.306748 24246 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.34.195:37423 every 8 connection(s)
I20260926 15:11:06.307945 24119 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435318278681-12427-0/minicluster-data/ts-2/data/info.pb
I20260926 15:11:06.310913 12427 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8uoJGv/build/asan/bin/kudu as pid 24119
I20260926 15:11:06.311074 12427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435318278681-12427-0/minicluster-data/ts-2/wal/instance
I20260926 15:11:06.311510 12427 external_mini_cluster.cc:1428] Setting key 1c43fa5ebd8a0df239a3b30cd2c68890
Sep 26 15:11:06 dist-test-slave-865v krb5kdc[23281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435466, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.195@KRBTEST.COM for kudu/127.12.34.254@KRBTEST.COM
I20260926 15:11:06.332649 24249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:11:06.310346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.195:33935 (local address 127.12.34.254:40719)
0926 15:11:06.310893 (+ 547us) server_negotiation.cc:206] Beginning negotiation
0926 15:11:06.310900 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:11:06.311913 (+ 1013us) server_negotiation.cc:407] Connection header received
0926 15:11:06.314351 (+ 2438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:11:06.314356 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:11:06.314458 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:11:06.314583 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:11:06.315888 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:11:06.316582 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:11:06.318083 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:11:06.318513 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:11:06.318647 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:11:06.327042 (+ 8395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:11:06.327048 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:11:06.327052 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:11:06.327090 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:11:06.329617 (+ 2527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:11:06.330370 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:11:06.330376 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:11:06.330379 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:11:06.330511 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:11:06.330940 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:11:06.330945 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:11:06.330948 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:11:06.331310 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:11:06.331451 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0926 15:11:06.332092 (+ 641us) server_negotiation.cc:299] Negotiation successful
0926 15:11:06.332296 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":244,"threads_started":1}
I20260926 15:11:06.333600 24250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:11:06.310877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:40719 (local address 127.12.34.195:33935)
0926 15:11:06.311748 (+ 871us) negotiation.cc:107] Waiting for socket to connect
0926 15:11:06.311796 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 15:11:06.314111 (+ 2315us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:11:06.314799 (+ 688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:11:06.314807 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:11:06.315204 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:11:06.315731 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:11:06.315747 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:11:06.316724 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:11:06.316731 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:11:06.317935 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:11:06.317946 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:11:06.318722 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:11:06.318729 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:11:06.318946 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:11:06.324473 (+ 5527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:11:06.324499 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:11:06.326808 (+ 2309us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:11:06.329750 (+ 2942us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:11:06.329759 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:11:06.329775 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:11:06.330254 (+ 479us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:11:06.330620 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:11:06.330625 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:11:06.330629 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:11:06.330824 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:11:06.331425 (+ 601us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:11:06.331433 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:11:06.331742 (+ 309us) client_negotiation.cc:769] Sending connection context
0926 15:11:06.331995 (+ 253us) client_negotiation.cc:240] Negotiation successful
0926 15:11:06.332466 (+ 471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":618,"thread_start_us":374,"threads_started":1}
I20260926 15:11:06.334865 24247 heartbeater.cc:347] Connected to a master server at 127.12.34.254:40719
I20260926 15:11:06.335120 24247 heartbeater.cc:464] Registering TS with master...
I20260926 15:11:06.335773 24247 heartbeater.cc:511] Master 127.12.34.254:40719 requested a full tablet report, sending...
I20260926 15:11:06.337143 23757 ts_manager.cc:194] Registered new tserver with Master: 3b9d7eb299a046398180424ee41b4e77 (127.12.34.195:37423)
I20260926 15:11:06.338115 23757 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.34.195@KRBTEST.COM'} at 127.12.34.195:33935
I20260926 15:11:06.339337 12427 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 15:11:06.356652 24249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:11:06.344121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42870 (local address 127.12.34.254:40719)
0926 15:11:06.344300 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0926 15:11:06.344304 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:11:06.344807 (+ 503us) server_negotiation.cc:407] Connection header received
0926 15:11:06.344929 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:11:06.344933 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:11:06.345003 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:11:06.345162 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:11:06.346535 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:11:06.347248 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:11:06.348775 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:11:06.349305 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:11:06.349408 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:11:06.351846 (+ 2438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:11:06.351850 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:11:06.351854 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:11:06.351881 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:11:06.354465 (+ 2584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:11:06.355105 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:11:06.355109 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:11:06.355111 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:11:06.355245 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:11:06.355567 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:11:06.355570 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:11:06.355572 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:11:06.355812 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:11:06.355976 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0926 15:11:06.356280 (+ 304us) server_negotiation.cc:299] Negotiation successful
0926 15:11:06.356434 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 15:11:06.360337 23757 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42870:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 15:11:06.367760 23757 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 15:11:06.401443 24263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:11:06.380674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36520 (local address 127.12.34.193:46797)
0926 15:11:06.386581 (+ 5907us) server_negotiation.cc:206] Beginning negotiation
0926 15:11:06.386594 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 15:11:06.391909 (+ 5315us) server_negotiation.cc:407] Connection header received
0926 15:11:06.393306 (+ 1397us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:11:06.393315 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:11:06.393522 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:11:06.393716 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:11:06.394513 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:11:06.396028 (+ 1515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:11:06.398579 (+ 2551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:11:06.400240 (+ 1661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:11:06.400374 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:11:06.400597 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0926 15:11:06.400891 (+ 294us) server_negotiation.cc:299] Negotiation successful
0926 15:11:06.401065 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5677,"thread_start_us":5524,"threads_started":1}
I20260926 15:11:06.401530 24262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:11:06.380094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.193:46797 (local address 127.0.0.1:36520)
0926 15:11:06.391731 (+ 11637us) negotiation.cc:107] Waiting for socket to connect
0926 15:11:06.391758 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0926 15:11:06.393101 (+ 1343us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:11:06.393702 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:11:06.393707 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:11:06.393735 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:11:06.394303 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:11:06.394316 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:11:06.396172 (+ 1856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:11:06.396178 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:11:06.398430 (+ 2252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:11:06.398441 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:11:06.400442 (+ 2001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:11:06.400448 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:11:06.400627 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:11:06.400685 (+ 58us) client_negotiation.cc:769] Sending connection context
0926 15:11:06.400822 (+ 137us) client_negotiation.cc:240] Negotiation successful
0926 15:11:06.401008 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11432,"mutex_wait_us":226,"thread_start_us":11845,"threads_started":1}
I20260926 15:11:06.401635 24264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:11:06.381353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.194:45599 (local address 127.0.0.1:46138)
0926 15:11:06.392030 (+ 10677us) negotiation.cc:107] Waiting for socket to connect
0926 15:11:06.392050 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 15:11:06.392579 (+ 529us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:11:06.393961 (+ 1382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:11:06.393966 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:11:06.393994 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:11:06.394528 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:11:06.394538 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:11:06.395911 (+ 1373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:11:06.395917 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:11:06.398928 (+ 3011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:11:06.398939 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:11:06.400778 (+ 1839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:11:06.400786 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:11:06.400965 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:11:06.400995 (+ 30us) client_negotiation.cc:769] Sending connection context
0926 15:11:06.401072 (+ 77us) client_negotiation.cc:240] Negotiation successful
0926 15:11:06.401224 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10549,"mutex_wait_us":142,"thread_start_us":10417,"threads_started":1}
I20260926 15:11:06.402050 24260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:11:06.379563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.195:37423 (local address 127.0.0.1:57926)
0926 15:11:06.380471 (+ 908us) negotiation.cc:107] Waiting for socket to connect
0926 15:11:06.380520 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 15:11:06.380693 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:11:06.393494 (+ 12801us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:11:06.393502 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:11:06.393530 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:11:06.394005 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:11:06.394020 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:11:06.396031 (+ 2011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:11:06.396035 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:11:06.398880 (+ 2845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:11:06.398898 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:11:06.401209 (+ 2311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:11:06.401216 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:11:06.401384 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:11:06.401419 (+ 35us) client_negotiation.cc:769] Sending connection context
0926 15:11:06.401517 (+ 98us) client_negotiation.cc:240] Negotiation successful
0926 15:11:06.401682 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":670,"mutex_wait_us":123,"thread_start_us":522,"threads_started":1}
I20260926 15:11:06.402246 24265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:11:06.381470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46138 (local address 127.12.34.194:45599)
0926 15:11:06.393023 (+ 11553us) server_negotiation.cc:206] Beginning negotiation
0926 15:11:06.393031 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 15:11:06.393053 (+ 22us) server_negotiation.cc:407] Connection header received
0926 15:11:06.393347 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:11:06.393360 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:11:06.393561 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:11:06.393752 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:11:06.394669 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:11:06.395749 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:11:06.399081 (+ 3332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:11:06.400335 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:11:06.400457 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:11:06.400643 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0926 15:11:06.401811 (+ 1168us) server_negotiation.cc:299] Negotiation successful
0926 15:11:06.401985 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11416,"thread_start_us":11311,"threads_started":1}
I20260926 15:11:06.402802 24261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:11:06.379950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57926 (local address 127.12.34.195:37423)
0926 15:11:06.392818 (+ 12868us) server_negotiation.cc:206] Beginning negotiation
0926 15:11:06.392827 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 15:11:06.392859 (+ 32us) server_negotiation.cc:407] Connection header received
0926 15:11:06.392954 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:11:06.392961 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:11:06.393256 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:11:06.394080 (+ 824us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:11:06.394326 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:11:06.395874 (+ 1548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:11:06.399047 (+ 3173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:11:06.400823 (+ 1776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:11:06.400938 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:11:06.402376 (+ 1438us) server_negotiation.cc:1035] Waiting for connection context
0926 15:11:06.402442 (+ 66us) server_negotiation.cc:299] Negotiation successful
0926 15:11:06.402551 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12685,"thread_start_us":13138,"threads_started":1}
I20260926 15:11:06.405925 24048 tablet_service.cc:1467] Processing CreateTablet for tablet e9ce696029a74355b3581ec5694fb5e7 (DEFAULT_TABLE table=test-table [id=444e247876354f349ab27f22f65d3772]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:11:06.406059 24182 tablet_service.cc:1467] Processing CreateTablet for tablet e9ce696029a74355b3581ec5694fb5e7 (DEFAULT_TABLE table=test-table [id=444e247876354f349ab27f22f65d3772]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:11:06.407219 24048 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9ce696029a74355b3581ec5694fb5e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:11:06.407452 24182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9ce696029a74355b3581ec5694fb5e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:11:06.407503 23905 tablet_service.cc:1467] Processing CreateTablet for tablet e9ce696029a74355b3581ec5694fb5e7 (DEFAULT_TABLE table=test-table [id=444e247876354f349ab27f22f65d3772]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:11:06.408823 23905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9ce696029a74355b3581ec5694fb5e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:11:06.418320 24266 tablet_bootstrap.cc:492] T e9ce696029a74355b3581ec5694fb5e7 P ae3c0a0842794b77ba9995dff32974a9: Bootstrap starting.
I20260926 15:11:06.421167 24267 tablet_bootstrap.cc:492] T e9ce696029a74355b3581ec5694fb5e7 P 3b9d7eb299a046398180424ee41b4e77: Bootstrap starting.
I20260926 15:11:06.423236 24266 tablet_bootstrap.cc:654] T e9ce696029a74355b3581ec5694fb5e7 P ae3c0a0842794b77ba9995dff32974a9: Neither blocks nor log segments found. Creating new log.
I20260926 15:11:06.424492 24267 tablet_bootstrap.cc:654] T e9ce696029a74355b3581ec5694fb5e7 P 3b9d7eb299a046398180424ee41b4e77: Neither blocks nor log segments found. Creating new log.
I20260926 15:11:06.424474 24266 log.cc:824] T e9ce696029a74355b3581ec5694fb5e7 P ae3c0a0842794b77ba9995dff32974a9: Log is configured to *not* fsync() on all Append() calls
I20260926 15:11:06.425678 24267 log.cc:824] T e9ce696029a74355b3581ec5694fb5e7 P 3b9d7eb299a046398180424ee41b4e77: Log is configured to *not* fsync() on all Append() calls
I20260926 15:11:06.426069 24268 tablet_bootstrap.cc:492] T e9ce696029a74355b3581ec5694fb5e7 P 85eed29572b54740a074baa22c71d446: Bootstrap starting.
I20260926 15:11:06.428398 24266 tablet_bootstrap.cc:492] T e9ce696029a74355b3581ec5694fb5e7 P ae3c0a0842794b77ba9995dff32974a9: No bootstrap required, opened a new log
I20260926 15:11:06.428761 24266 ts_tablet_manager.cc:1397] T e9ce696029a74355b3581ec5694fb5e7 P ae3c0a0842794b77ba9995dff32974a9: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260926 15:11:06.431625 24268 tablet_bootstrap.cc:654] T e9ce696029a74355b3581ec5694fb5e7 P 85eed29572b54740a074baa22c71d446: Neither blocks nor log segments found. Creating new log.
I20260926 15:11:06.431989 24267 tablet_bootstrap.cc:492] T e9ce696029a74355b3581ec5694fb5e7 P 3b9d7eb299a046398180424ee41b4e77: No bootstrap required, opened a new log
I20260926 15:11:06.432242 24267 ts_tablet_manager.cc:1397] T e9ce696029a74355b3581ec5694fb5e7 P 3b9d7eb299a046398180424ee41b4e77: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260926 15:11:06.432636 24268 log.cc:824] T e9ce696029a74355b3581ec5694fb5e7 P 85eed29572b54740a074baa22c71d446: Log is configured to *not* fsync() on all Append() calls
I20260926 15:11:06.435357 24268 tablet_bootstrap.cc:492] T e9ce696029a74355b3581ec5694fb5e7 P 85eed29572b54740a074baa22c71d446: No bootstrap required, opened a new log
I20260926 15:11:06.435595 24268 ts_tablet_manager.cc:1397] T e9ce696029a74355b3581ec5694fb5e7 P 85eed29572b54740a074baa22c71d446: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.003s
I20260926 15:11:06.439414 24266 raft_consensus.cc:348] T e9ce696029a74355b3581ec5694fb5e7 P ae3c0a0842794b77ba9995dff32974a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b9d7eb299a046398180424ee41b4e77" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 37423 } } peers { permanent_uuid: "ae3c0a0842794b77ba9995dff32974a9" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 45599 } } peers { permanent_uuid: "85eed29572b54740a074baa22c71d446" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 46797 } }
I20260926 15:11:06.439817 24266 raft_consensus.cc:374] T e9ce696029a74355b3581ec5694fb5e7 P ae3c0a0842794b77ba9995dff32974a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:11:06.439922 24266 raft_consensus.cc:729] T e9ce696029a74355b3581ec5694fb5e7 P ae3c0a0842794b77ba9995dff32974a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae3c0a0842794b77ba9995dff32974a9, State: Initialized, Role: FOLLOWER
I20260926 15:11:06.440606 24266 consensus_queue.cc:260] T e9ce696029a74355b3581ec5694fb5e7 P ae3c0a0842794b77ba9995dff32974a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b9d7eb299a046398180424ee41b4e77" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 37423 } } peers { permanent_uuid: "ae3c0a0842794b77ba9995dff32974a9" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 45599 } } peers { permanent_uuid: "85eed29572b54740a074baa22c71d446" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 46797 } }
I20260926 15:11:06.442289 24113 heartbeater.cc:503] Master 127.12.34.254:40719 was elected leader, sending a full tablet report...
I20260926 15:11:06.443265 24266 ts_tablet_manager.cc:1428] T e9ce696029a74355b3581ec5694fb5e7 P ae3c0a0842794b77ba9995dff32974a9: Time spent starting tablet: real 0.014s user 0.009s sys 0.004s
I20260926 15:11:06.444106 24268 raft_consensus.cc:348] T e9ce696029a74355b3581ec5694fb5e7 P 85eed29572b54740a074baa22c71d446 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b9d7eb299a046398180424ee41b4e77" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 37423 } } peers { permanent_uuid: "ae3c0a0842794b77ba9995dff32974a9" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 45599 } } peers { permanent_uuid: "85eed29572b54740a074baa22c71d446" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 46797 } }
I20260926 15:11:06.444116 24267 raft_consensus.cc:348] T e9ce696029a74355b3581ec5694fb5e7 P 3b9d7eb299a046398180424ee41b4e77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b9d7eb299a046398180424ee41b4e77" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 37423 } } peers { permanent_uuid: "ae3c0a0842794b77ba9995dff32974a9" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 45599 } } peers { permanent_uuid: "85eed29572b54740a074baa22c71d446" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 46797 } }
I20260926 15:11:06.444425 24268 raft_consensus.cc:374] T e9ce696029a74355b3581ec5694fb5e7 P 85eed29572b54740a074baa22c71d446 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:11:06.444519 24267 raft_consensus.cc:374] T e9ce696029a74355b3581ec5694fb5e7 P 3b9d7eb299a046398180424ee41b4e77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:11:06.444550 24268 raft_consensus.cc:729] T e9ce696029a74355b3581ec5694fb5e7 P 85eed29572b54740a074baa22c71d446 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85eed29572b54740a074baa22c71d446, State: Initialized, Role: FOLLOWER
I20260926 15:11:06.444620 24267 raft_consensus.cc:729] T e9ce696029a74355b3581ec5694fb5e7 P 3b9d7eb299a046398180424ee41b4e77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b9d7eb299a046398180424ee41b4e77, State: Initialized, Role: FOLLOWER
I20260926 15:11:06.445232 24268 consensus_queue.cc:260] T e9ce696029a74355b3581ec5694fb5e7 P 85eed29572b54740a074baa22c71d446 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b9d7eb299a046398180424ee41b4e77" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 37423 } } peers { permanent_uuid: "ae3c0a0842794b77ba9995dff32974a9" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 45599 } } peers { permanent_uuid: "85eed29572b54740a074baa22c71d446" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 46797 } }
I20260926 15:11:06.445356 24267 consensus_queue.cc:260] T e9ce696029a74355b3581ec5694fb5e7 P 3b9d7eb299a046398180424ee41b4e77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b9d7eb299a046398180424ee41b4e77" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 37423 } } peers { permanent_uuid: "ae3c0a0842794b77ba9995dff32974a9" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 45599 } } peers { permanent_uuid: "85eed29572b54740a074baa22c71d446" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 46797 } }
I20260926 15:11:06.446902 24268 ts_tablet_manager.cc:1428] T e9ce696029a74355b3581ec5694fb5e7 P 85eed29572b54740a074baa22c71d446: Time spent starting tablet: real 0.011s user 0.008s sys 0.004s
I20260926 15:11:06.447847 24247 heartbeater.cc:503] Master 127.12.34.254:40719 was elected leader, sending a full tablet report...
I20260926 15:11:06.448503 24267 ts_tablet_manager.cc:1428] T e9ce696029a74355b3581ec5694fb5e7 P 3b9d7eb299a046398180424ee41b4e77: Time spent starting tablet: real 0.016s user 0.005s sys 0.011s
W20260926 15:11:06.473286 24114 tablet.cc:2369] T e9ce696029a74355b3581ec5694fb5e7 P ae3c0a0842794b77ba9995dff32974a9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 15:11:06.559508 24248 tablet.cc:2369] T e9ce696029a74355b3581ec5694fb5e7 P 3b9d7eb299a046398180424ee41b4e77: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 15:11:06.569912 24274 raft_consensus.cc:482] T e9ce696029a74355b3581ec5694fb5e7 P 3b9d7eb299a046398180424ee41b4e77 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 15:11:06.570312 24274 raft_consensus.cc:504] T e9ce696029a74355b3581ec5694fb5e7 P 3b9d7eb299a046398180424ee41b4e77 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b9d7eb299a046398180424ee41b4e77" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 37423 } } peers { permanent_uuid: "ae3c0a0842794b77ba9995dff32974a9" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 45599 } } peers { permanent_uuid: "85eed29572b54740a074baa22c71d446" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 46797 } }
I20260926 15:11:06.574051 24274 leader_election.cc:288] T e9ce696029a74355b3581ec5694fb5e7 P 3b9d7eb299a046398180424ee41b4e77 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae3c0a0842794b77ba9995dff32974a9 (127.12.34.194:45599), 85eed29572b54740a074baa22c71d446 (127.12.34.193:46797)
I20260926 15:11:06.579401 24265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:11:06.573142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.195:52975 (local address 127.12.34.194:45599)
0926 15:11:06.573329 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0926 15:11:06.573336 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:11:06.573359 (+ 23us) server_negotiation.cc:407] Connection header received
0926 15:11:06.573565 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:11:06.573575 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:11:06.573674 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:11:06.573781 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:11:06.574340 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:11:06.575364 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:11:06.577419 (+ 2055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:11:06.578571 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:11:06.578681 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:11:06.578731 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 15:11:06.579105 (+ 374us) server_negotiation.cc:299] Negotiation successful
0926 15:11:06.579219 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 15:11:06.579488 24250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:11:06.572723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.194:45599 (local address 127.12.34.195:52975)
0926 15:11:06.573202 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0926 15:11:06.573221 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 15:11:06.573353 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:11:06.573953 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:11:06.573958 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:11:06.573984 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:11:06.574211 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:11:06.574223 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:11:06.575586 (+ 1363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:11:06.575591 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:11:06.577284 (+ 1693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:11:06.577295 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:11:06.578730 (+ 1435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:11:06.578736 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:11:06.578911 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:11:06.578967 (+ 56us) client_negotiation.cc:769] Sending connection context
0926 15:11:06.579068 (+ 101us) client_negotiation.cc:240] Negotiation successful
0926 15:11:06.579199 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"mutex_wait_us":5}
I20260926 15:11:06.580567 24068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9ce696029a74355b3581ec5694fb5e7" candidate_uuid: "3b9d7eb299a046398180424ee41b4e77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae3c0a0842794b77ba9995dff32974a9" is_pre_election: true
I20260926 15:11:06.580788 24263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:11:06.573162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.195:58279 (local address 127.12.34.193:46797)
0926 15:11:06.573749 (+ 587us) server_negotiation.cc:206] Beginning negotiation
0926 15:11:06.573757 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 15:11:06.574110 (+ 353us) server_negotiation.cc:407] Connection header received
0926 15:11:06.574328 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:11:06.574333 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:11:06.574407 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:11:06.574472 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:11:06.575169 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:11:06.576289 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:11:06.578379 (+ 2090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:11:06.579472 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:11:06.579650 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:11:06.579720 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 15:11:06.580415 (+ 695us) server_negotiation.cc:299] Negotiation successful
0926 15:11:06.580548 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 15:11:06.580809 24275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:11:06.572831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.193:46797 (local address 127.12.34.195:58279)
0926 15:11:06.574007 (+ 1176us) negotiation.cc:107] Waiting for socket to connect
0926 15:11:06.574020 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 15:11:06.574176 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:11:06.574658 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:11:06.574663 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:11:06.574687 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:11:06.574999 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:11:06.575009 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:11:06.576470 (+ 1461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:11:06.576474 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:11:06.578219 (+ 1745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:11:06.578234 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:11:06.580102 (+ 1868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:11:06.580110 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:11:06.580260 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:11:06.580281 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 15:11:06.580383 (+ 102us) client_negotiation.cc:240] Negotiation successful
0926 15:11:06.580495 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1080,"thread_start_us":906,"threads_started":1}
I20260926 15:11:06.581152 24068 raft_consensus.cc:2454] T e9ce696029a74355b3581ec5694fb5e7 P ae3c0a0842794b77ba9995dff32974a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b9d7eb299a046398180424ee41b4e77 in term 0.
I20260926 15:11:06.581707 23925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9ce696029a74355b3581ec5694fb5e7" candidate_uuid: "3b9d7eb299a046398180424ee41b4e77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85eed29572b54740a074baa22c71d446" is_pre_election: true
I20260926 15:11:06.581856 24136 leader_election.cc:302] T e9ce696029a74355b3581ec5694fb5e7 P 3b9d7eb299a046398180424ee41b4e77 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b9d7eb299a046398180424ee41b4e77, ae3c0a0842794b77ba9995dff32974a9; no voters:
I20260926 15:11:06.582212 24274 raft_consensus.cc:2781] T e9ce696029a74355b3581ec5694fb5e7 P 3b9d7eb299a046398180424ee41b4e77 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 15:11:06.582276 23925 raft_consensus.cc:2454] T e9ce696029a74355b3581ec5694fb5e7 P 85eed29572b54740a074baa22c71d446 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b9d7eb299a046398180424ee41b4e77 in term 0.
I20260926 15:11:06.582363 24274 raft_consensus.cc:482] T e9ce696029a74355b3581ec5694fb5e7 P 3b9d7eb299a046398180424ee41b4e77 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 15:11:06.582458 24274 raft_consensus.cc:3037] T e9ce696029a74355b3581ec5694fb5e7 P 3b9d7eb299a046398180424ee41b4e77 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:11:06.585047 24274 raft_consensus.cc:504] T e9ce696029a74355b3581ec5694fb5e7 P 3b9d7eb299a046398180424ee41b4e77 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b9d7eb299a046398180424ee41b4e77" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 37423 } } peers { permanent_uuid: "ae3c0a0842794b77ba9995dff32974a9" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 45599 } } peers { permanent_uuid: "85eed29572b54740a074baa22c71d446" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 46797 } }
I20260926 15:11:06.585623 24274 leader_election.cc:288] T e9ce696029a74355b3581ec5694fb5e7 P 3b9d7eb299a046398180424ee41b4e77 [CANDIDATE]: Term 1 election: Requested vote from peers ae3c0a0842794b77ba9995dff32974a9 (127.12.34.194:45599), 85eed29572b54740a074baa22c71d446 (127.12.34.193:46797)
I20260926 15:11:06.586117 23925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9ce696029a74355b3581ec5694fb5e7" candidate_uuid: "3b9d7eb299a046398180424ee41b4e77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85eed29572b54740a074baa22c71d446"
I20260926 15:11:06.586319 23925 raft_consensus.cc:3037] T e9ce696029a74355b3581ec5694fb5e7 P 85eed29572b54740a074baa22c71d446 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:11:06.588478 23925 raft_consensus.cc:2454] T e9ce696029a74355b3581ec5694fb5e7 P 85eed29572b54740a074baa22c71d446 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b9d7eb299a046398180424ee41b4e77 in term 1.
I20260926 15:11:06.588872 24135 leader_election.cc:302] T e9ce696029a74355b3581ec5694fb5e7 P 3b9d7eb299a046398180424ee41b4e77 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b9d7eb299a046398180424ee41b4e77, 85eed29572b54740a074baa22c71d446; no voters:
I20260926 15:11:06.588843 24068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9ce696029a74355b3581ec5694fb5e7" candidate_uuid: "3b9d7eb299a046398180424ee41b4e77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae3c0a0842794b77ba9995dff32974a9"
I20260926 15:11:06.589071 24068 raft_consensus.cc:3037] T e9ce696029a74355b3581ec5694fb5e7 P ae3c0a0842794b77ba9995dff32974a9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:11:06.589190 24274 raft_consensus.cc:2781] T e9ce696029a74355b3581ec5694fb5e7 P 3b9d7eb299a046398180424ee41b4e77 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:11:06.589673 24274 raft_consensus.cc:686] T e9ce696029a74355b3581ec5694fb5e7 P 3b9d7eb299a046398180424ee41b4e77 [term 1 LEADER]: Becoming Leader. State: Replica: 3b9d7eb299a046398180424ee41b4e77, State: Running, Role: LEADER
I20260926 15:11:06.590173 24274 consensus_queue.cc:237] T e9ce696029a74355b3581ec5694fb5e7 P 3b9d7eb299a046398180424ee41b4e77 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b9d7eb299a046398180424ee41b4e77" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 37423 } } peers { permanent_uuid: "ae3c0a0842794b77ba9995dff32974a9" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 45599 } } peers { permanent_uuid: "85eed29572b54740a074baa22c71d446" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 46797 } }
I20260926 15:11:06.592114 24068 raft_consensus.cc:2454] T e9ce696029a74355b3581ec5694fb5e7 P ae3c0a0842794b77ba9995dff32974a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b9d7eb299a046398180424ee41b4e77 in term 1.
I20260926 15:11:06.594825 23757 catalog_manager.cc:5495] T e9ce696029a74355b3581ec5694fb5e7 P 3b9d7eb299a046398180424ee41b4e77 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3b9d7eb299a046398180424ee41b4e77 (127.12.34.195). New cstate: current_term: 1 leader_uuid: "3b9d7eb299a046398180424ee41b4e77" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b9d7eb299a046398180424ee41b4e77" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 37423 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae3c0a0842794b77ba9995dff32974a9" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 45599 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "85eed29572b54740a074baa22c71d446" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 46797 } health_report { overall_health: UNKNOWN } } }
I20260926 15:11:06.639505 24261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:11:06.634056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57936 (local address 127.12.34.195:37423)
0926 15:11:06.634216 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0926 15:11:06.634221 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:11:06.634237 (+ 16us) server_negotiation.cc:407] Connection header received
0926 15:11:06.634414 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:11:06.634418 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:11:06.634487 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:11:06.634607 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 15:11:06.634929 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:11:06.635698 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:11:06.637181 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:11:06.637940 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:11:06.638035 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:11:06.638534 (+ 499us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 15:11:06.639017 (+ 483us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 15:11:06.639129 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 15:11:06.639250 (+ 121us) server_negotiation.cc:299] Negotiation successful
0926 15:11:06.639312 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 15:11:06.647439 24068 raft_consensus.cc:1261] T e9ce696029a74355b3581ec5694fb5e7 P ae3c0a0842794b77ba9995dff32974a9 [term 1 FOLLOWER]: Refusing update from remote peer 3b9d7eb299a046398180424ee41b4e77: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:11:06.647440 23925 raft_consensus.cc:1261] T e9ce696029a74355b3581ec5694fb5e7 P 85eed29572b54740a074baa22c71d446 [term 1 FOLLOWER]: Refusing update from remote peer 3b9d7eb299a046398180424ee41b4e77: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:11:06.648458 24274 consensus_queue.cc:1035] T e9ce696029a74355b3581ec5694fb5e7 P 3b9d7eb299a046398180424ee41b4e77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae3c0a0842794b77ba9995dff32974a9" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 45599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:11:06.648746 24276 consensus_queue.cc:1035] T e9ce696029a74355b3581ec5694fb5e7 P 3b9d7eb299a046398180424ee41b4e77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85eed29572b54740a074baa22c71d446" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 46797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:11:06.672178 23757 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42870:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 15:11:06.672430 23757 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42870:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 15:11:06.676117 23757 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dee3b6427b5e4aab8a67763313094ee5: Sending DeleteTablet for 3 replicas of tablet e9ce696029a74355b3581ec5694fb5e7
I20260926 15:11:06.677565 24182 tablet_service.cc:1514] Processing DeleteTablet for tablet e9ce696029a74355b3581ec5694fb5e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:11:06 UTC) from {username='kudu', principal='kudu/127.12.34.254@KRBTEST.COM'} at 127.0.0.1:57926
I20260926 15:11:06.677680 24048 tablet_service.cc:1514] Processing DeleteTablet for tablet e9ce696029a74355b3581ec5694fb5e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:11:06 UTC) from {username='kudu', principal='kudu/127.12.34.254@KRBTEST.COM'} at 127.0.0.1:46138
I20260926 15:11:06.677683 23905 tablet_service.cc:1514] Processing DeleteTablet for tablet e9ce696029a74355b3581ec5694fb5e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:11:06 UTC) from {username='kudu', principal='kudu/127.12.34.254@KRBTEST.COM'} at 127.0.0.1:36520
I20260926 15:11:06.680037 12427 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 23837
I20260926 15:11:06.690689 24288 tablet_replica.cc:330] T e9ce696029a74355b3581ec5694fb5e7 P 3b9d7eb299a046398180424ee41b4e77: stopping tablet replica
I20260926 15:11:06.691008 24290 tablet_replica.cc:330] T e9ce696029a74355b3581ec5694fb5e7 P 85eed29572b54740a074baa22c71d446: stopping tablet replica
I20260926 15:11:06.691579 24289 tablet_replica.cc:330] T e9ce696029a74355b3581ec5694fb5e7 P ae3c0a0842794b77ba9995dff32974a9: stopping tablet replica
I20260926 15:11:06.691571 24288 raft_consensus.cc:2229] T e9ce696029a74355b3581ec5694fb5e7 P 3b9d7eb299a046398180424ee41b4e77 [term 1 LEADER]: Raft consensus shutting down.
I20260926 15:11:06.691970 24289 raft_consensus.cc:2229] T e9ce696029a74355b3581ec5694fb5e7 P ae3c0a0842794b77ba9995dff32974a9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 15:11:06.692426 24290 raft_consensus.cc:2229] T e9ce696029a74355b3581ec5694fb5e7 P 85eed29572b54740a074baa22c71d446 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 15:11:06.692512 24288 raft_consensus.cc:2258] T e9ce696029a74355b3581ec5694fb5e7 P 3b9d7eb299a046398180424ee41b4e77 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:11:06.693008 24289 raft_consensus.cc:2258] T e9ce696029a74355b3581ec5694fb5e7 P ae3c0a0842794b77ba9995dff32974a9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:11:06.693532 24290 raft_consensus.cc:2258] T e9ce696029a74355b3581ec5694fb5e7 P 85eed29572b54740a074baa22c71d446 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:11:06.695006 24288 ts_tablet_manager.cc:1905] T e9ce696029a74355b3581ec5694fb5e7 P 3b9d7eb299a046398180424ee41b4e77: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:11:06.696961 24289 ts_tablet_manager.cc:1905] T e9ce696029a74355b3581ec5694fb5e7 P ae3c0a0842794b77ba9995dff32974a9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:11:06.696725 24290 ts_tablet_manager.cc:1905] T e9ce696029a74355b3581ec5694fb5e7 P 85eed29572b54740a074baa22c71d446: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 26 15:11:06 dist-test-slave-865v krb5kdc[23281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435462, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.34.193@KRBTEST.COM
I20260926 15:11:06.702911 24288 ts_tablet_manager.cc:1918] T e9ce696029a74355b3581ec5694fb5e7 P 3b9d7eb299a046398180424ee41b4e77: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 15:11:06.703190 24288 log.cc:1196] T e9ce696029a74355b3581ec5694fb5e7 P 3b9d7eb299a046398180424ee41b4e77: Deleting WAL directory at /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435318278681-12427-0/minicluster-data/ts-2/wal/wals/e9ce696029a74355b3581ec5694fb5e7
I20260926 15:11:06.703799 24288 ts_tablet_manager.cc:1939] T e9ce696029a74355b3581ec5694fb5e7 P 3b9d7eb299a046398180424ee41b4e77: Deleting consensus metadata
I20260926 15:11:06.705673 23742 catalog_manager.cc:4841] TS 3b9d7eb299a046398180424ee41b4e77 (127.12.34.195:37423): tablet e9ce696029a74355b3581ec5694fb5e7 (table test-table [id=444e247876354f349ab27f22f65d3772]) successfully deleted
I20260926 15:11:06.706475 24290 ts_tablet_manager.cc:1918] T e9ce696029a74355b3581ec5694fb5e7 P 85eed29572b54740a074baa22c71d446: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 15:11:06.706732 24290 log.cc:1196] T e9ce696029a74355b3581ec5694fb5e7 P 85eed29572b54740a074baa22c71d446: Deleting WAL directory at /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435318278681-12427-0/minicluster-data/ts-0/wal/wals/e9ce696029a74355b3581ec5694fb5e7
I20260926 15:11:06.707334 24290 ts_tablet_manager.cc:1939] T e9ce696029a74355b3581ec5694fb5e7 P 85eed29572b54740a074baa22c71d446: Deleting consensus metadata
I20260926 15:11:06.709144 23741 catalog_manager.cc:4841] TS 85eed29572b54740a074baa22c71d446 (127.12.34.193:46797): tablet e9ce696029a74355b3581ec5694fb5e7 (table test-table [id=444e247876354f349ab27f22f65d3772]) successfully deleted
I20260926 15:11:06.710829 24289 ts_tablet_manager.cc:1918] T e9ce696029a74355b3581ec5694fb5e7 P ae3c0a0842794b77ba9995dff32974a9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 15:11:06.711053 24289 log.cc:1196] T e9ce696029a74355b3581ec5694fb5e7 P ae3c0a0842794b77ba9995dff32974a9: Deleting WAL directory at /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790435318278681-12427-0/minicluster-data/ts-1/wal/wals/e9ce696029a74355b3581ec5694fb5e7
I20260926 15:11:06.711989 24263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:11:06.683189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36528 (local address 127.12.34.193:46797)
0926 15:11:06.683370 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0926 15:11:06.683376 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:11:06.687837 (+ 4461us) server_negotiation.cc:407] Connection header received
0926 15:11:06.688057 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:11:06.688063 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:11:06.688160 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:11:06.688268 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:11:06.692821 (+ 4553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:11:06.694066 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:11:06.697175 (+ 3109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:11:06.697843 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:11:06.697953 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:11:06.703568 (+ 5615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:11:06.703584 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:11:06.703602 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:11:06.703650 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:11:06.707790 (+ 4140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:11:06.708711 (+ 921us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:11:06.708721 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:11:06.708725 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:11:06.708892 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:11:06.709813 (+ 921us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:11:06.709819 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:11:06.709822 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:11:06.710249 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:11:06.710524 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0926 15:11:06.710902 (+ 378us) server_negotiation.cc:299] Negotiation successful
0926 15:11:06.711162 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 15:11:06.712028 24289 ts_tablet_manager.cc:1939] T e9ce696029a74355b3581ec5694fb5e7 P ae3c0a0842794b77ba9995dff32974a9: Deleting consensus metadata
I20260926 15:11:06.712716 23965 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:11:06.713923 23742 catalog_manager.cc:4841] TS ae3c0a0842794b77ba9995dff32974a9 (127.12.34.194:45599): tablet e9ce696029a74355b3581ec5694fb5e7 (table test-table [id=444e247876354f349ab27f22f65d3772]) successfully deleted
I20260926 15:11:06.854166 12427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8uoJGv/build/asan/bin/kudu with pid 23837
I20260926 15:11:06.873097 12427 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 23976
Sep 26 15:11:06 dist-test-slave-865v krb5kdc[23281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435462, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.34.194@KRBTEST.COM
I20260926 15:11:06.891659 24265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:11:06.874172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46154 (local address 127.12.34.194:45599)
0926 15:11:06.874387 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0926 15:11:06.874394 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:11:06.874417 (+ 23us) server_negotiation.cc:407] Connection header received
0926 15:11:06.874487 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:11:06.874494 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:11:06.874596 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:11:06.874715 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:11:06.875911 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:11:06.877219 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:11:06.878566 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:11:06.879262 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:11:06.879406 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:11:06.884391 (+ 4985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:11:06.884410 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:11:06.884429 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:11:06.884476 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:11:06.888410 (+ 3934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:11:06.889134 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:11:06.889144 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:11:06.889148 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:11:06.889304 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:11:06.889734 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:11:06.889739 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:11:06.889743 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:11:06.890206 (+ 463us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:11:06.890479 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0926 15:11:06.890894 (+ 415us) server_negotiation.cc:299] Negotiation successful
0926 15:11:06.891263 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260926 15:11:06.892433 24108 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:11:06.982990 12427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8uoJGv/build/asan/bin/kudu with pid 23976
I20260926 15:11:06.999840 12427 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 24119
Sep 26 15:11:07 dist-test-slave-865v krb5kdc[23281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435462, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.34.195@KRBTEST.COM
I20260926 15:11:07.015611 24261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:11:07.000627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57946 (local address 127.12.34.195:37423)
0926 15:11:07.000824 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0926 15:11:07.000831 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:11:07.000853 (+ 22us) server_negotiation.cc:407] Connection header received
0926 15:11:07.001007 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:11:07.001012 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:11:07.001140 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:11:07.001258 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:11:07.002445 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:11:07.003200 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:11:07.004465 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:11:07.004905 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:11:07.005066 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:11:07.009879 (+ 4813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:11:07.009890 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:11:07.009905 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:11:07.009950 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:11:07.012901 (+ 2951us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:11:07.013587 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:11:07.013599 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:11:07.013601 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:11:07.013742 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:11:07.014105 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:11:07.014111 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:11:07.014114 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:11:07.014453 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:11:07.014773 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0926 15:11:07.015056 (+ 283us) server_negotiation.cc:299] Negotiation successful
0926 15:11:07.015276 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260926 15:11:07.016337 24242 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:11:07.137216 12427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8uoJGv/build/asan/bin/kudu with pid 24119
I20260926 15:11:07.156240 12427 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 23725
I20260926 15:11:07.157567 23787 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:11:07.266911 12427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8uoJGv/build/asan/bin/kudu with pid 23725
I20260926 15:11:07.288774 12427 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260926 15:11:07.623469 12427 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260926 15:11:07.623719 12427 mini_ranger.cc:66] Stopping Ranger...
I20260926 15:11:07.968636 12427 mini_ranger.cc:68] Stopped Ranger
2026-09-26 15:11:07.968 UTC [23311] LOG: received smart shutdown request
2026-09-26 15:11:07.972 UTC [23311] LOG: background worker "logical replication launcher" (PID 23320) exited with exit code 1
2026-09-26 15:11:07.973 UTC [23315] LOG: shutting down
2026-09-26 15:11:08.190 UTC [23311] LOG: database system is shut down
2026-09-26T15:11:08Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (49614 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 15:11:08 dist-test-slave-865v krb5kdc[24304](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 15:11:08 dist-test-slave-865v krb5kdc[24304](info): set up 2 sockets
Sep 26 15:11:08 dist-test-slave-865v krb5kdc[24304](info): commencing operation
krb5kdc: starting...
W20260926 15:11:09.449457 12427 mini_kdc.cc:121] Time spent starting KDC: real 1.084s user 0.002s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/krb5kdc/test-user.keytab.
Sep 26 15:11:09 dist-test-slave-865v krb5kdc[24304](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435469, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T15:11:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T15:11:09Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.12.34.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.12.34.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.12.34.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/krb5kdc/rangeradmin_127.12.34.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.12.34.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/krb5kdc/rangeradmin_127.12.34.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.12.34.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.12.34.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.12.34.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/krb5kdc/rangerlookup_127.12.34.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.12.34.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/krb5kdc/rangerlookup_127.12.34.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.12.34.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.34.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.34.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/krb5kdc/HTTP_127.12.34.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.12.34.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/krb5kdc/HTTP_127.12.34.212@KRBTEST.COM.keytab.
I20260926 15:11:09.768704 12427 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task8uoJGv/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/minicluster-data/postgres -l logfile start
2026-09-26 15:11:12.497 UTC [24334] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-26 15:11:12.497 UTC [24334] LOG: listening on IPv4 address "127.12.34.212", port 46859
2026-09-26 15:11:12.501 UTC [24334] LOG: listening on Unix socket "/tmp/.s.PGSQL.46859"
2026-09-26 15:11:12.523 UTC [24337] LOG: database system was shut down at 2026-09-26 15:11:10 UTC
2026-09-26 15:11:12.527 UTC [24334] LOG: database system is ready to accept connections
I20260926 15:11:13.505326 12427 mini_postgres.cc:96] Postgres bound to 46859
2026-09-26 15:11:13.522 UTC [24345] FATAL: database "slave" does not exist
127.12.34.212:46859 - accepting connections
I20260926 15:11:13.522707 12427 mini_ranger.cc:161] Starting Ranger...
I20260926 15:11:13.549685 12427 mini_ranger.cc:84] Created miniranger Postgres user
I20260926 15:11:14.110208 12427 mini_ranger.cc:87] Created ranger Postgres database
I20260926 15:11:14.110327 12427 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/minicluster-data/ranger-admin
2026-09-26 15:11:14,392 [I] DB FLAVOR :POSTGRES
2026-09-26 15:11:14,393 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 15:11:14,393 [I] Checking connection..
2026-09-26 15:11:14,393 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8uoJGv/build/asan/bin/postgresql.jar:/tmp/dist-test-task8uoJGv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.34.212:46859/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-26 15:11:14,637 [I] Checking connection passed.
2026-09-26 15:11:14,637 [I] --------- Verifying version history table ---------
2026-09-26 15:11:14,637 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8uoJGv/build/asan/bin/postgresql.jar:/tmp/dist-test-task8uoJGv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.34.212:46859/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 15:11:14,899 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 15:11:14,899 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8uoJGv/build/asan/bin/postgresql.jar:/tmp/dist-test-task8uoJGv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.34.212:46859/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 15:11:15,126 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 15:11:15,126 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-26 15:11:15,126 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8uoJGv/build/asan/bin/postgresql.jar:/tmp/dist-test-task8uoJGv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.34.212:46859/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task8uoJGv/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-26 15:11:15.373 UTC [24409] WARNING: there is no transaction in progress
2026-09-26 15:11:15,380 [I] create_dbversion_catalog.sql file imported successfully
2026-09-26 15:11:15,380 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8uoJGv/build/asan/bin/postgresql.jar:/tmp/dist-test-task8uoJGv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.34.212:46859/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 15:11:15,628 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-26 15:11:15,628 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 15:11:15,628 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8uoJGv/build/asan/bin/postgresql.jar:/tmp/dist-test-task8uoJGv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.34.212:46859/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-26 15:11:15,867 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8uoJGv/build/asan/bin/postgresql.jar:/tmp/dist-test-task8uoJGv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.34.212:46859/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-26 15:11:16,094 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8uoJGv/build/asan/bin/postgresql.jar:/tmp/dist-test-task8uoJGv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.34.212:46859/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-865v.c.gcp-upstream.internal','N') ;"
2026-09-26 15:11:16,321 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8uoJGv/build/asan/bin/postgresql.jar:/tmp/dist-test-task8uoJGv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.34.212:46859/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 15:11:16,579 [I] Table x_portal_user does not exist in database ranger
2026-09-26 15:11:16,579 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8uoJGv/build/asan/bin/postgresql.jar:/tmp/dist-test-task8uoJGv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.34.212:46859/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 15:11:16,824 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-26 15:11:16,824 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8uoJGv/build/asan/bin/postgresql.jar:/tmp/dist-test-task8uoJGv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.34.212:46859/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 15:11:17,094 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8uoJGv/build/asan/bin/postgresql.jar:/tmp/dist-test-task8uoJGv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.34.212:46859/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 15:11:17,334 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-26 15:11:17,335 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8uoJGv/build/asan/bin/postgresql.jar:/tmp/dist-test-task8uoJGv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.34.212:46859/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task8uoJGv/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-26 15:11:18.296 UTC [24517] WARNING: there is no transaction in progress
2026-09-26 15:11:18.312 UTC [24517] WARNING: there is no transaction in progress
2026-09-26 15:11:18.329 UTC [24517] WARNING: there is no transaction in progress
2026-09-26 15:11:18.344 UTC [24517] WARNING: there is no transaction in progress
2026-09-26 15:11:18.360 UTC [24517] WARNING: there is no transaction in progress
2026-09-26 15:11:18.377 UTC [24517] WARNING: there is no transaction in progress
2026-09-26 15:11:18.429 UTC [24517] WARNING: there is no transaction in progress
2026-09-26 15:11:18.436 UTC [24517] WARNING: there is no transaction in progress
2026-09-26 15:11:18.452 UTC [24517] WARNING: there is no transaction in progress
2026-09-26 15:11:18.462 UTC [24517] WARNING: there is no transaction in progress
2026-09-26 15:11:18.472 UTC [24517] WARNING: there is no transaction in progress
2026-09-26 15:11:18.485 UTC [24517] WARNING: there is no transaction in progress
2026-09-26 15:11:18.494 UTC [24517] WARNING: there is no transaction in progress
2026-09-26 15:11:18.501 UTC [24517] WARNING: there is no transaction in progress
2026-09-26 15:11:18.509 UTC [24517] WARNING: there is no transaction in progress
2026-09-26 15:11:18.516 UTC [24517] WARNING: there is no transaction in progress
2026-09-26 15:11:19,202 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-26 15:11:19,202 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8uoJGv/build/asan/bin/postgresql.jar:/tmp/dist-test-task8uoJGv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.34.212:46859/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-26 15:11:19,452 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-26 15:11:19,453 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8uoJGv/build/asan/bin/postgresql.jar:/tmp/dist-test-task8uoJGv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.34.212:46859/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 15:11:19,710 [I] Table x_portal_user already exists in database 'ranger'
2026-09-26 15:11:19,711 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8uoJGv/build/asan/bin/postgresql.jar:/tmp/dist-test-task8uoJGv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.34.212:46859/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 15:11:19,937 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-26 15:11:19,937 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8uoJGv/build/asan/bin/postgresql.jar:/tmp/dist-test-task8uoJGv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.34.212:46859/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 15:11:20,196 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8uoJGv/build/asan/bin/postgresql.jar:/tmp/dist-test-task8uoJGv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.34.212:46859/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 15:11:20,462 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8uoJGv/build/asan/bin/postgresql.jar:/tmp/dist-test-task8uoJGv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.34.212:46859/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-865v.c.gcp-upstream.internal';"
2026-09-26 15:11:20,687 [I] CORE_DB_SCHEMA import status has been updated
2026-09-26 15:11:20,688 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8uoJGv/build/asan/bin/postgresql.jar:/tmp/dist-test-task8uoJGv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.34.212:46859/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-26 15:11:20,957 [I] DB_PATCHES have already been applied
I20260926 15:11:20.963577 12427 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/minicluster-data/ranger-admin:/tmp/dist-test-task8uoJGv/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task8uoJGv/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task8uoJGv/build/asan/bin/postgresql.jar:/tmp/dist-test-task8uoJGv/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260926 15:11:20.963660 12427 mini_ranger.cc:193] Using host: 127.12.34.212
I20260926 15:11:20.978037 12427 mini_ranger.cc:227] Ranger admin URL: http://127.12.34.212:55605
Sep 26, 2026 3:11:21 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 3:11:21 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 3:11:21 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/minicluster-data/ranger-admin/ews/webapp
Sep 26, 2026 3:11:21 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 26, 2026 3:11:21 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 26, 2026 3:11:21 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/minicluster-data/ranger-admin/ews/webapp].
Sep 26, 2026 3:11:21 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260926 15:11:22.095952 12427 mini_ranger.cc:160] Time spent starting Ranger: real 8.573s user 0.000s sys 0.037s
Sep 26, 2026 3:11:41 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 26, 2026 3:11:41 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 26, 2026 3:11:41 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 26, 2026 3:11:41 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 26, 2026 3:11:41 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 26, 2026 3:11:41 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 26, 2026 3:11:41 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 26, 2026 3:11:41 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 26, 2026 3:11:41 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 26, 2026 3:11:41 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 26, 2026 3:11:41 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 26, 2026 3:11:41 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 26, 2026 3:11:41 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 26, 2026 3:11:41 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 26, 2026 3:11:41 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 26, 2026 3:11:41 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 26, 2026 3:11:41 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 26, 2026 3:11:42 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260926 15:11:42.643363 12427 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.12.34.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.12.34.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.12.34.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/krb5kdc/rangerkms_127.12.34.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.12.34.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/krb5kdc/rangerkms_127.12.34.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.12.34.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.12.34.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.34.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/krb5kdc/HTTP_127.12.34.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.12.34.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/krb5kdc/HTTP_127.12.34.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 26 15:11:42 dist-test-slave-865v krb5kdc[24304](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435502, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260926 15:11:42.820528 12427 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260926 15:11:42.841658 12427 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260926 15:11:43.776396 12427 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260926 15:11:43.776518 12427 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/minicluster-data/ranger-kms
I20260926 15:11:43.776563 12427 mini_ranger_kms.cc:227] Using postgres at 127.12.34.212:46859
2026-09-26 15:11:44,112 [I] DB FLAVOR :POSTGRES
2026-09-26 15:11:44,112 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 15:11:44,112 [I] Checking connection
2026-09-26 15:11:44,113 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8uoJGv/build/asan/bin/postgresql.jar:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.34.212:46859/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-26 15:11:44,357 [I] connection success
2026-09-26 15:11:44,358 [I] --------- Verifying Ranger DB tables ---------
2026-09-26 15:11:44,358 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-26 15:11:44,358 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8uoJGv/build/asan/bin/postgresql.jar:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.34.212:46859/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-26 15:11:44,581 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-26 15:11:44,581 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 15:11:44,581 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-26 15:11:44,581 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8uoJGv/build/asan/bin/postgresql.jar:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.34.212:46859/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-26 15:11:44,843 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260926 15:11:45.278069 12427 mini_ranger_kms.cc:310] Created kms service
I20260926 15:11:45.486624 12427 mini_ranger_kms.cc:326] Created kudu user
I20260926 15:11:45.528939 12427 mini_ranger_kms.cc:343] Created rangerkms user
Sep 26 15:11:45 dist-test-slave-865v krb5kdc[24304](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435502, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.12.34.212@KRBTEST.COM
I20260926 15:11:45.919766 12427 mini_ranger_kms.cc:384] Added ranger policy
I20260926 15:11:45.920284 12427 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/minicluster-data/ranger-kms:/tmp/dist-test-task8uoJGv/build/asan/bin/postgresql.jar:/tmp/dist-test-task8uoJGv/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task8uoJGv/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260926 15:11:45.920353 12427 mini_ranger_kms.cc:240] Using host: 127.12.34.212
I20260926 15:11:45.937805 12427 mini_ranger_kms.cc:276] Ranger KMS PID: 24697
I20260926 15:11:45.937934 12427 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.12.34.212:60253
Sep 26, 2026 3:11:47 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 3:11:47 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 3:11:47 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 26, 2026 3:11:47 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 26, 2026 3:11:47 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/minicluster-data/ranger-kms/ews/webapp].
Sep 26, 2026 3:11:47 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260926 15:11:48.241173 12427 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.421s user 0.062s sys 0.289s
I20260926 15:11:48.241425 12427 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260926 15:11:48.241616 12427 mini_ranger_kms.cc:401] 127.12.34.212:60253/kms/v1/keys
Sep 26 15:11:48 dist-test-slave-865v krb5kdc[24304](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790435508, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.12.34.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 26 15:11:48 dist-test-slave-865v krb5kdc[24304](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435508, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.12.34.212@KRBTEST.COM for HTTP/127.12.34.212@KRBTEST.COM
Sep 26 15:11:49 dist-test-slave-865v krb5kdc[24304](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435508, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.12.34.212@KRBTEST.COM for HTTP/127.12.34.212@KRBTEST.COM
Sep 26 15:11:51 dist-test-slave-865v krb5kdc[24304](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435511, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.12.34.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.34.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.34.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.254.keytab.
Entry for principal kudu/127.12.34.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.254.keytab.
I20260926 15:11:51.716941 12427 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8uoJGv/build/asan/bin/kudu
/tmp/dist-test-task8uoJGv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.34.254:45857
--webserver_interface=127.12.34.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.254.keytab
--principal=kudu/127.12.34.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.34.212:45919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.34.254:45857
--ranger_config_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.34.212:60253/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:11:52.053587 24748 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:11:52.053876 24748 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:11:52.054039 24748 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:11:52.061620 24748 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:11:52.061718 24748 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:11:52.061807 24748 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:11:52.061882 24748 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:11:52.061946 24748 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:11:52.072177 24748 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.34.212:45919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/minicluster-data/master-0/wal
--ranger_kms_url=127.12.34.212:60253/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.12.34.254:45857
--ranger_config_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.254.keytab
--principal=kudu/127.12.34.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.34.254:45857
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.34.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:11:52.073550 24748 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:11:52.074906 24748 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:11:52.087242 24753 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:11:52.087282 24754 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:11:52.087306 24756 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:11:52.087642 24748 server_base.cc:1034] running on GCE node
I20260926 15:11:52.088653 24748 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:11:52.090057 24748 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:11:52.091288 24748 hybrid_clock.cc:648] HybridClock initialized: now 1790435512091257 us; error 49 us; skew 500 ppm
I20260926 15:11:52.091629 24748 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:11:52 dist-test-slave-865v krb5kdc[24304](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435512, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:11:52.094741 24748 init.cc:377] Logged in from keytab as kudu/127.12.34.254@KRBTEST.COM (short username kudu)
I20260926 15:11:52.096176 24748 webserver.cc:460] Webserver started at http://127.12.34.254:36207/ using document root <none> and password file <none>
I20260926 15:11:52.096802 24748 fs_manager.cc:359] Metadata directory not provided
I20260926 15:11:52.096937 24748 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:11:52.097296 24748 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 15:11:52 dist-test-slave-865v krb5kdc[24304](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435512, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.254@KRBTEST.COM for HTTP/127.12.34.212@KRBTEST.COM
I20260926 15:11:52.159894 24748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/minicluster-data/master-0/data/instance:
uuid: "503ca1bbe012419898d9ac8c1fa73872"
format_stamp: "Formatted at 2026-09-26 15:11:52 on dist-test-slave-865v"
server_key: "87e47af19dc8c5f65fc38de7cc316758"
server_key_iv: "5efb5d478a82fda602c96bdf507d5000"
server_key_version: "kuduclusterkey@0"
I20260926 15:11:52.160770 24748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/minicluster-data/master-0/wal/instance:
uuid: "503ca1bbe012419898d9ac8c1fa73872"
format_stamp: "Formatted at 2026-09-26 15:11:52 on dist-test-slave-865v"
server_key: "87e47af19dc8c5f65fc38de7cc316758"
server_key_iv: "5efb5d478a82fda602c96bdf507d5000"
server_key_version: "kuduclusterkey@0"
I20260926 15:11:52.166435 24748 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260926 15:11:52.186473 24763 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:11:52.188208 24748 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260926 15:11:52.188405 24748 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/minicluster-data/master-0/data,/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/minicluster-data/master-0/wal
uuid: "503ca1bbe012419898d9ac8c1fa73872"
format_stamp: "Formatted at 2026-09-26 15:11:52 on dist-test-slave-865v"
server_key: "87e47af19dc8c5f65fc38de7cc316758"
server_key_iv: "5efb5d478a82fda602c96bdf507d5000"
server_key_version: "kuduclusterkey@0"
I20260926 15:11:52.188702 24748 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:11:52.214262 24748 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:11:52.218464 24748 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:11:52.218775 24748 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:11:52.238119 24748 rpc_server.cc:307] RPC server started. Bound to: 127.12.34.254:45857
I20260926 15:11:52.238122 24814 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.34.254:45857 every 8 connection(s)
I20260926 15:11:52.239423 24748 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
I20260926 15:11:52.243089 24815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:11:52.243853 12427 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8uoJGv/build/asan/bin/kudu as pid 24748
I20260926 15:11:52.244077 12427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/minicluster-data/master-0/wal/instance
I20260926 15:11:52.244915 12427 external_mini_cluster.cc:1428] Setting key adce50dbb7e2efdc75e9a7cde61b4d72
I20260926 15:11:52.255725 24815 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 503ca1bbe012419898d9ac8c1fa73872: Bootstrap starting.
Sep 26 15:11:52 dist-test-slave-865v krb5kdc[24304](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435511, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.34.254@KRBTEST.COM
I20260926 15:11:52.258976 24815 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 503ca1bbe012419898d9ac8c1fa73872: Neither blocks nor log segments found. Creating new log.
I20260926 15:11:52.259974 24815 log.cc:824] T 00000000000000000000000000000000 P 503ca1bbe012419898d9ac8c1fa73872: Log is configured to *not* fsync() on all Append() calls
I20260926 15:11:52.262981 24815 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 503ca1bbe012419898d9ac8c1fa73872: No bootstrap required, opened a new log
I20260926 15:11:52.266665 24818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:11:52.247138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43498 (local address 127.12.34.254:45857)
0926 15:11:52.247934 (+ 796us) server_negotiation.cc:206] Beginning negotiation
0926 15:11:52.247945 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 15:11:52.247983 (+ 38us) server_negotiation.cc:407] Connection header received
0926 15:11:52.249723 (+ 1740us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:11:52.249744 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:11:52.250109 (+ 365us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:11:52.250494 (+ 385us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:11:52.251814 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:11:52.253206 (+ 1392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:11:52.254545 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:11:52.255189 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:11:52.255375 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:11:52.259511 (+ 4136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:11:52.259519 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:11:52.259533 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:11:52.259566 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:11:52.262857 (+ 3291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:11:52.263521 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:11:52.263536 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:11:52.263541 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:11:52.263700 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:11:52.264074 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:11:52.264080 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:11:52.264083 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:11:52.264564 (+ 481us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:11:52.264869 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0926 15:11:52.265349 (+ 480us) server_negotiation.cc:299] Negotiation successful
0926 15:11:52.265716 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":599,"thread_start_us":372,"threads_started":1}
I20260926 15:11:52.270835 24815 raft_consensus.cc:348] T 00000000000000000000000000000000 P 503ca1bbe012419898d9ac8c1fa73872 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "503ca1bbe012419898d9ac8c1fa73872" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 45857 } }
I20260926 15:11:52.271183 24815 raft_consensus.cc:374] T 00000000000000000000000000000000 P 503ca1bbe012419898d9ac8c1fa73872 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:11:52.271286 24815 raft_consensus.cc:729] T 00000000000000000000000000000000 P 503ca1bbe012419898d9ac8c1fa73872 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 503ca1bbe012419898d9ac8c1fa73872, State: Initialized, Role: FOLLOWER
I20260926 15:11:52.271869 24815 consensus_queue.cc:260] T 00000000000000000000000000000000 P 503ca1bbe012419898d9ac8c1fa73872 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "503ca1bbe012419898d9ac8c1fa73872" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 45857 } }
I20260926 15:11:52.272166 24815 raft_consensus.cc:388] T 00000000000000000000000000000000 P 503ca1bbe012419898d9ac8c1fa73872 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 15:11:52.272328 24815 raft_consensus.cc:482] T 00000000000000000000000000000000 P 503ca1bbe012419898d9ac8c1fa73872 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 15:11:52.272522 24815 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 503ca1bbe012419898d9ac8c1fa73872 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:11:52.274681 24815 raft_consensus.cc:504] T 00000000000000000000000000000000 P 503ca1bbe012419898d9ac8c1fa73872 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "503ca1bbe012419898d9ac8c1fa73872" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 45857 } }
I20260926 15:11:52.275098 24815 leader_election.cc:302] T 00000000000000000000000000000000 P 503ca1bbe012419898d9ac8c1fa73872 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 503ca1bbe012419898d9ac8c1fa73872; no voters:
I20260926 15:11:52.275585 24815 leader_election.cc:288] T 00000000000000000000000000000000 P 503ca1bbe012419898d9ac8c1fa73872 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 15:11:52.275739 24820 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 503ca1bbe012419898d9ac8c1fa73872 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:11:52.276666 24820 raft_consensus.cc:686] T 00000000000000000000000000000000 P 503ca1bbe012419898d9ac8c1fa73872 [term 1 LEADER]: Becoming Leader. State: Replica: 503ca1bbe012419898d9ac8c1fa73872, State: Running, Role: LEADER
I20260926 15:11:52.277289 24820 consensus_queue.cc:237] T 00000000000000000000000000000000 P 503ca1bbe012419898d9ac8c1fa73872 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "503ca1bbe012419898d9ac8c1fa73872" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 45857 } }
I20260926 15:11:52.278236 24815 sys_catalog.cc:564] T 00000000000000000000000000000000 P 503ca1bbe012419898d9ac8c1fa73872 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:11:52.280134 24821 sys_catalog.cc:455] T 00000000000000000000000000000000 P 503ca1bbe012419898d9ac8c1fa73872 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "503ca1bbe012419898d9ac8c1fa73872" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "503ca1bbe012419898d9ac8c1fa73872" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 45857 } } }
I20260926 15:11:52.280669 24821 sys_catalog.cc:458] T 00000000000000000000000000000000 P 503ca1bbe012419898d9ac8c1fa73872 [sys.catalog]: This master's current role is: LEADER
I20260926 15:11:52.281847 24828 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:11:52.280857 24822 sys_catalog.cc:455] T 00000000000000000000000000000000 P 503ca1bbe012419898d9ac8c1fa73872 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 503ca1bbe012419898d9ac8c1fa73872. Latest consensus state: current_term: 1 leader_uuid: "503ca1bbe012419898d9ac8c1fa73872" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "503ca1bbe012419898d9ac8c1fa73872" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 45857 } } }
I20260926 15:11:52.283519 24822 sys_catalog.cc:458] T 00000000000000000000000000000000 P 503ca1bbe012419898d9ac8c1fa73872 [sys.catalog]: This master's current role is: LEADER
I20260926 15:11:52.287658 24828 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 15:11:52.288403 24815 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260926 15:11:52.307376 24828 catalog_manager.cc:1347] Generated new cluster ID: 5187116344214de2a22feb8eb50ee8aa
I20260926 15:11:52.307535 24828 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 15:11:52.323057 24828 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 15:11:52.325169 24828 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:11:52.333904 24828 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 503ca1bbe012419898d9ac8c1fa73872: Generated new TSK 0
I20260926 15:11:52.334964 24828 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.34.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.34.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.34.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.193.keytab.
Entry for principal kudu/127.12.34.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.193.keytab.
I20260926 15:11:53.094020 12427 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8uoJGv/build/asan/bin/kudu
/tmp/dist-test-task8uoJGv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.34.212:60253/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.34.193:0
--local_ip_for_outbound_sockets=127.12.34.193
--webserver_interface=127.12.34.193
--webserver_port=0
--tserver_master_addrs=127.12.34.254:45857
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.193.keytab
--principal=kudu/127.12.34.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.34.212:45919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 15:11:53 dist-test-slave-865v krb5kdc[24304](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790435513, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260926 15:11:53.538656 24860 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:11:53.539044 24860 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:11:53.539256 24860 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:11:53.550805 24860 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:11:53.550931 24860 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:11:53.551050 24860 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.34.193
I20260926 15:11:53.561942 24860 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.34.212:45919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.12.34.212:60253/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.193.keytab
--principal=kudu/127.12.34.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.34.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.34.193
--webserver_port=0
--tserver_master_addrs=127.12.34.254:45857
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.34.193
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:11:53.563535 24860 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:11:53.564981 24860 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:11:53.597141 24870 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:11:53.598037 24871 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:11:53.606508 24873 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:11:53.606917 24860 server_base.cc:1034] running on GCE node
I20260926 15:11:53.607635 24860 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:11:53.608799 24860 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:11:53.610028 24860 hybrid_clock.cc:648] HybridClock initialized: now 1790435513609973 us; error 46 us; skew 500 ppm
I20260926 15:11:53.610401 24860 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:11:53 dist-test-slave-865v krb5kdc[24304](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435513, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:11:53.613951 24860 init.cc:377] Logged in from keytab as kudu/127.12.34.193@KRBTEST.COM (short username kudu)
I20260926 15:11:53.615916 24860 webserver.cc:460] Webserver started at http://127.12.34.193:38989/ using document root <none> and password file <none>
I20260926 15:11:53.617318 24860 fs_manager.cc:359] Metadata directory not provided
I20260926 15:11:53.617466 24860 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:11:53.617820 24860 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 15:11:53 dist-test-slave-865v krb5kdc[24304](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435513, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.193@KRBTEST.COM for HTTP/127.12.34.212@KRBTEST.COM
I20260926 15:11:53.648041 24860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/minicluster-data/ts-0/data/instance:
uuid: "a26ca453e2c941d2b4d5e9972af91fde"
format_stamp: "Formatted at 2026-09-26 15:11:53 on dist-test-slave-865v"
server_key: "daf77363c027870de5bd2476911c363f"
server_key_iv: "6dfe77c37baca6ef60e01abf9aa1759a"
server_key_version: "kuduclusterkey@0"
I20260926 15:11:53.649350 24860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/minicluster-data/ts-0/wal/instance:
uuid: "a26ca453e2c941d2b4d5e9972af91fde"
format_stamp: "Formatted at 2026-09-26 15:11:53 on dist-test-slave-865v"
server_key: "daf77363c027870de5bd2476911c363f"
server_key_iv: "6dfe77c37baca6ef60e01abf9aa1759a"
server_key_version: "kuduclusterkey@0"
I20260926 15:11:53.657147 24860 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s
I20260926 15:11:53.704216 24880 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:11:53.706393 24860 fs_manager.cc:725] Time spent opening block manager: real 0.018s user 0.009s sys 0.008s
I20260926 15:11:53.706604 24860 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/minicluster-data/ts-0/data,/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/minicluster-data/ts-0/wal
uuid: "a26ca453e2c941d2b4d5e9972af91fde"
format_stamp: "Formatted at 2026-09-26 15:11:53 on dist-test-slave-865v"
server_key: "daf77363c027870de5bd2476911c363f"
server_key_iv: "6dfe77c37baca6ef60e01abf9aa1759a"
server_key_version: "kuduclusterkey@0"
I20260926 15:11:53.706926 24860 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:11:53.771740 24860 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:11:53.778689 24860 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:11:53.779089 24860 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:11:53.793638 24860 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:11:53.805958 24860 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:11:53.806121 24860 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260926 15:11:53.806314 24860 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:11:53.806434 24860 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 15:11:53 dist-test-slave-865v krb5kdc[24304](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435513, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.254@KRBTEST.COM for HTTP/127.12.34.212@KRBTEST.COM
I20260926 15:11:54.209376 24860 rpc_server.cc:307] RPC server started. Bound to: 127.12.34.193:33971
I20260926 15:11:54.209408 24992 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.34.193:33971 every 8 connection(s)
I20260926 15:11:54.210704 24860 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/minicluster-data/ts-0/data/info.pb
I20260926 15:11:54.215945 12427 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8uoJGv/build/asan/bin/kudu as pid 24860
I20260926 15:11:54.216205 12427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/minicluster-data/ts-0/wal/instance
I20260926 15:11:54.217244 12427 external_mini_cluster.cc:1428] Setting key f0dd5949ea0dad27cf970e5cbb361c15
WARNING: no policy specified for kudu/127.12.34.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.34.194@KRBTEST.COM" created.
Sep 26 15:11:54 dist-test-slave-865v krb5kdc[24304](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435513, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.193@KRBTEST.COM for kudu/127.12.34.254@KRBTEST.COM
I20260926 15:11:54.258773 24996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:11:54.218233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.193:33999 (local address 127.12.34.254:45857)
0926 15:11:54.237500 (+ 19267us) server_negotiation.cc:206] Beginning negotiation
0926 15:11:54.237505 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:11:54.237526 (+ 21us) server_negotiation.cc:407] Connection header received
0926 15:11:54.237634 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:11:54.237639 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:11:54.237745 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:11:54.239985 (+ 2240us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:11:54.240154 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:11:54.241293 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:11:54.243530 (+ 2237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:11:54.244174 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:11:54.246809 (+ 2635us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:11:54.250386 (+ 3577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:11:54.250393 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:11:54.250398 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:11:54.250446 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:11:54.254090 (+ 3644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:11:54.255067 (+ 977us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:11:54.255073 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:11:54.255076 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:11:54.255222 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:11:54.256392 (+ 1170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:11:54.256396 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:11:54.256399 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:11:54.256732 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:11:54.256862 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0926 15:11:54.258226 (+ 1364us) server_negotiation.cc:299] Negotiation successful
0926 15:11:54.258430 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19133,"thread_start_us":18998,"threads_started":1}
I20260926 15:11:54.259549 24995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:11:54.214844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:45857 (local address 127.12.34.193:33999)
0926 15:11:54.221731 (+ 6887us) negotiation.cc:107] Waiting for socket to connect
0926 15:11:54.221786 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0926 15:11:54.224665 (+ 2879us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:11:54.238256 (+ 13591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:11:54.238270 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:11:54.238937 (+ 667us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:11:54.239775 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:11:54.239800 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:11:54.241482 (+ 1682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:11:54.241488 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:11:54.243334 (+ 1846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:11:54.243349 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:11:54.244422 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:11:54.244429 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:11:54.244689 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:11:54.245496 (+ 807us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:11:54.245525 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:11:54.250095 (+ 4570us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:11:54.254255 (+ 4160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:11:54.254265 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:11:54.254282 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:11:54.254902 (+ 620us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:11:54.255348 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:11:54.255353 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:11:54.255356 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:11:54.255667 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:11:54.256899 (+ 1232us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:11:54.256907 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:11:54.257376 (+ 469us) client_negotiation.cc:769] Sending connection context
0926 15:11:54.257695 (+ 319us) client_negotiation.cc:240] Negotiation successful
0926 15:11:54.258205 (+ 510us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6519,"thread_start_us":6257,"threads_started":1}
I20260926 15:11:54.261687 24993 heartbeater.cc:347] Connected to a master server at 127.12.34.254:45857
I20260926 15:11:54.261997 24993 heartbeater.cc:464] Registering TS with master...
I20260926 15:11:54.262825 24993 heartbeater.cc:511] Master 127.12.34.254:45857 requested a full tablet report, sending...
I20260926 15:11:54.265431 24780 ts_manager.cc:194] Registered new tserver with Master: a26ca453e2c941d2b4d5e9972af91fde (127.12.34.193:33971)
I20260926 15:11:54.267638 24780 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.34.193@KRBTEST.COM'} at 127.12.34.193:33999
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.34.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.194.keytab.
Entry for principal kudu/127.12.34.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.194.keytab.
I20260926 15:11:54.285231 12427 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8uoJGv/build/asan/bin/kudu
/tmp/dist-test-task8uoJGv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.34.212:60253/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.34.194:0
--local_ip_for_outbound_sockets=127.12.34.194
--webserver_interface=127.12.34.194
--webserver_port=0
--tserver_master_addrs=127.12.34.254:45857
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.194.keytab
--principal=kudu/127.12.34.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.34.212:45919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 15:11:54 dist-test-slave-865v krb5kdc[24304](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435513, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.254@KRBTEST.COM for HTTP/127.12.34.212@KRBTEST.COM
W20260926 15:11:54.735867 24999 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:11:54.736244 24999 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:11:54.736457 24999 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:11:54.748186 24999 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:11:54.748342 24999 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:11:54.748575 24999 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.34.194
I20260926 15:11:54.764240 24999 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.34.212:45919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.12.34.212:60253/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.194.keytab
--principal=kudu/127.12.34.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.34.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.34.194
--webserver_port=0
--tserver_master_addrs=127.12.34.254:45857
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.34.194
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:11:54.766103 24999 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:11:54.767786 24999 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:11:54.785583 25013 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:11:54.786311 25014 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:11:54.795356 25016 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:11:54.796190 24999 server_base.cc:1034] running on GCE node
I20260926 15:11:54.796933 24999 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:11:54.798148 24999 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:11:54.799427 24999 hybrid_clock.cc:648] HybridClock initialized: now 1790435514799366 us; error 45 us; skew 500 ppm
I20260926 15:11:54.799784 24999 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:11:54 dist-test-slave-865v krb5kdc[24304](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435514, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:11:54.803678 24999 init.cc:377] Logged in from keytab as kudu/127.12.34.194@KRBTEST.COM (short username kudu)
I20260926 15:11:54.805537 24999 webserver.cc:460] Webserver started at http://127.12.34.194:38227/ using document root <none> and password file <none>
I20260926 15:11:54.806452 24999 fs_manager.cc:359] Metadata directory not provided
I20260926 15:11:54.806584 24999 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:11:54.806922 24999 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 15:11:54 dist-test-slave-865v krb5kdc[24304](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435514, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.194@KRBTEST.COM for HTTP/127.12.34.212@KRBTEST.COM
I20260926 15:11:54.827797 24999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/minicluster-data/ts-1/data/instance:
uuid: "ed30be0b68124b0ead8c15e6ffe0aef7"
format_stamp: "Formatted at 2026-09-26 15:11:54 on dist-test-slave-865v"
server_key: "c29c29b048fec2032595a9e88c75e89f"
server_key_iv: "d8e1631c29166075a8f42f365e931179"
server_key_version: "kuduclusterkey@0"
I20260926 15:11:54.828825 24999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/minicluster-data/ts-1/wal/instance:
uuid: "ed30be0b68124b0ead8c15e6ffe0aef7"
format_stamp: "Formatted at 2026-09-26 15:11:54 on dist-test-slave-865v"
server_key: "c29c29b048fec2032595a9e88c75e89f"
server_key_iv: "d8e1631c29166075a8f42f365e931179"
server_key_version: "kuduclusterkey@0"
I20260926 15:11:54.835794 24999 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20260926 15:11:54.853039 25023 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:11:54.854805 24999 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260926 15:11:54.855020 24999 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/minicluster-data/ts-1/data,/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/minicluster-data/ts-1/wal
uuid: "ed30be0b68124b0ead8c15e6ffe0aef7"
format_stamp: "Formatted at 2026-09-26 15:11:54 on dist-test-slave-865v"
server_key: "c29c29b048fec2032595a9e88c75e89f"
server_key_iv: "d8e1631c29166075a8f42f365e931179"
server_key_version: "kuduclusterkey@0"
I20260926 15:11:54.855309 24999 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:11:54.881698 24999 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:11:54.886775 24999 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:11:54.887168 24999 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:11:54.888633 24999 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:11:54.890175 24999 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:11:54.890277 24999 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:11:54.890429 24999 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:11:54.890499 24999 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:11:54.925710 24999 rpc_server.cc:307] RPC server started. Bound to: 127.12.34.194:34699
I20260926 15:11:54.925735 25135 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.34.194:34699 every 8 connection(s)
I20260926 15:11:54.927131 24999 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/minicluster-data/ts-1/data/info.pb
I20260926 15:11:54.928812 12427 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8uoJGv/build/asan/bin/kudu as pid 24999
I20260926 15:11:54.928956 12427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/minicluster-data/ts-1/wal/instance
I20260926 15:11:54.929517 12427 external_mini_cluster.cc:1428] Setting key e8b6039a62d4e8290fbf83c2a65fc2b5
Sep 26 15:11:54 dist-test-slave-865v krb5kdc[24304](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435514, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.194@KRBTEST.COM for kudu/127.12.34.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.12.34.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.34.195@KRBTEST.COM" created.
I20260926 15:11:54.969800 25138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:11:54.930234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.194:46623 (local address 127.12.34.254:45857)
0926 15:11:54.945470 (+ 15236us) server_negotiation.cc:206] Beginning negotiation
0926 15:11:54.945476 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:11:54.949640 (+ 4164us) server_negotiation.cc:407] Connection header received
0926 15:11:54.951473 (+ 1833us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:11:54.951479 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:11:54.951571 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:11:54.951687 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:11:54.953118 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:11:54.953877 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:11:54.955520 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:11:54.955927 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:11:54.956065 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:11:54.959443 (+ 3378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:11:54.959448 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:11:54.959452 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:11:54.959483 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:11:54.962477 (+ 2994us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:11:54.963365 (+ 888us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:11:54.963370 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:11:54.963374 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:11:54.963522 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:11:54.964033 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:11:54.964038 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:11:54.964041 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:11:54.964410 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:11:54.964555 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0926 15:11:54.969242 (+ 4687us) server_negotiation.cc:299] Negotiation successful
0926 15:11:54.969458 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15083,"thread_start_us":21263,"threads_started":1}
I20260926 15:11:54.970677 25139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:11:54.930806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:45857 (local address 127.12.34.194:46623)
0926 15:11:54.949448 (+ 18642us) negotiation.cc:107] Waiting for socket to connect
0926 15:11:54.949496 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 15:11:54.951198 (+ 1702us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:11:54.951930 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:11:54.951940 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:11:54.952368 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:11:54.952919 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:11:54.952934 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:11:54.954042 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:11:54.954050 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:11:54.955355 (+ 1305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:11:54.955370 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:11:54.956176 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:11:54.956184 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:11:54.956459 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:11:54.957199 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:11:54.957225 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:11:54.959218 (+ 1993us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:11:54.962621 (+ 3403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:11:54.962629 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:11:54.962644 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:11:54.963239 (+ 595us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:11:54.963663 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:11:54.963671 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:11:54.963678 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:11:54.963912 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:11:54.967217 (+ 3305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:11:54.967225 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:11:54.967645 (+ 420us) client_negotiation.cc:769] Sending connection context
0926 15:11:54.967939 (+ 294us) client_negotiation.cc:240] Negotiation successful
0926 15:11:54.968440 (+ 501us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18362,"thread_start_us":14712,"threads_started":1}
I20260926 15:11:54.972270 25136 heartbeater.cc:347] Connected to a master server at 127.12.34.254:45857
I20260926 15:11:54.972604 25136 heartbeater.cc:464] Registering TS with master...
I20260926 15:11:54.973457 25136 heartbeater.cc:511] Master 127.12.34.254:45857 requested a full tablet report, sending...
I20260926 15:11:54.975243 24780 ts_manager.cc:194] Registered new tserver with Master: ed30be0b68124b0ead8c15e6ffe0aef7 (127.12.34.194:34699)
I20260926 15:11:54.976363 24780 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.34.194@KRBTEST.COM'} at 127.12.34.194:46623
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.34.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.195.keytab.
Entry for principal kudu/127.12.34.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.195.keytab.
I20260926 15:11:54.992610 12427 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8uoJGv/build/asan/bin/kudu
/tmp/dist-test-task8uoJGv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.34.212:60253/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.34.195:0
--local_ip_for_outbound_sockets=127.12.34.195
--webserver_interface=127.12.34.195
--webserver_port=0
--tserver_master_addrs=127.12.34.254:45857
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.195.keytab
--principal=kudu/127.12.34.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.34.212:45919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 15:11:55.271260 24993 heartbeater.cc:503] Master 127.12.34.254:45857 was elected leader, sending a full tablet report...
W20260926 15:11:55.316524 25142 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:11:55.316766 25142 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:11:55.316857 25142 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:11:55.323874 25142 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:11:55.323969 25142 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:11:55.324075 25142 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.34.195
I20260926 15:11:55.333418 25142 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.34.212:45919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.12.34.212:60253/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.195.keytab
--principal=kudu/127.12.34.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.34.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.34.195
--webserver_port=0
--tserver_master_addrs=127.12.34.254:45857
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.34.195
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:11:55.334481 25142 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:11:55.335592 25142 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:11:55.348001 25148 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:11:55.348349 25150 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:11:55.348295 25147 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:11:55.349620 25142 server_base.cc:1034] running on GCE node
I20260926 15:11:55.350190 25142 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:11:55.351086 25142 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:11:55.352319 25142 hybrid_clock.cc:648] HybridClock initialized: now 1790435515352273 us; error 41 us; skew 500 ppm
I20260926 15:11:55.352634 25142 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:11:55 dist-test-slave-865v krb5kdc[24304](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435515, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:11:55.355696 25142 init.cc:377] Logged in from keytab as kudu/127.12.34.195@KRBTEST.COM (short username kudu)
I20260926 15:11:55.356916 25142 webserver.cc:460] Webserver started at http://127.12.34.195:41741/ using document root <none> and password file <none>
I20260926 15:11:55.357547 25142 fs_manager.cc:359] Metadata directory not provided
I20260926 15:11:55.357649 25142 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:11:55.357880 25142 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 15:11:55 dist-test-slave-865v krb5kdc[24304](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435515, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.195@KRBTEST.COM for HTTP/127.12.34.212@KRBTEST.COM
I20260926 15:11:55.371052 25142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/minicluster-data/ts-2/data/instance:
uuid: "5933f4a56ce04a8f9cc6c35ec5e7e243"
format_stamp: "Formatted at 2026-09-26 15:11:55 on dist-test-slave-865v"
server_key: "f784631f0d2979c1cac701e8716732a8"
server_key_iv: "65eb352b56df397fc0e739d8d7c341cc"
server_key_version: "kuduclusterkey@0"
I20260926 15:11:55.371850 25142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/minicluster-data/ts-2/wal/instance:
uuid: "5933f4a56ce04a8f9cc6c35ec5e7e243"
format_stamp: "Formatted at 2026-09-26 15:11:55 on dist-test-slave-865v"
server_key: "f784631f0d2979c1cac701e8716732a8"
server_key_iv: "65eb352b56df397fc0e739d8d7c341cc"
server_key_version: "kuduclusterkey@0"
I20260926 15:11:55.377063 25142 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260926 15:11:55.394426 25157 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:11:55.395845 25142 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 15:11:55.396016 25142 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/minicluster-data/ts-2/data,/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/minicluster-data/ts-2/wal
uuid: "5933f4a56ce04a8f9cc6c35ec5e7e243"
format_stamp: "Formatted at 2026-09-26 15:11:55 on dist-test-slave-865v"
server_key: "f784631f0d2979c1cac701e8716732a8"
server_key_iv: "65eb352b56df397fc0e739d8d7c341cc"
server_key_version: "kuduclusterkey@0"
I20260926 15:11:55.396207 25142 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:11:55.431871 25142 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:11:55.435474 25142 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:11:55.435715 25142 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:11:55.436568 25142 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:11:55.437850 25142 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:11:55.437919 25142 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:11:55.438000 25142 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:11:55.438051 25142 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:11:55.470667 25142 rpc_server.cc:307] RPC server started. Bound to: 127.12.34.195:33371
I20260926 15:11:55.470712 25269 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.34.195:33371 every 8 connection(s)
I20260926 15:11:55.472023 25142 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/minicluster-data/ts-2/data/info.pb
I20260926 15:11:55.474200 12427 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8uoJGv/build/asan/bin/kudu as pid 25142
I20260926 15:11:55.474371 12427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/minicluster-data/ts-2/wal/instance
I20260926 15:11:55.474946 12427 external_mini_cluster.cc:1428] Setting key ddae4935270353ebe0ed2bc25b4d1882
Sep 26 15:11:55 dist-test-slave-865v krb5kdc[24304](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435515, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.195@KRBTEST.COM for kudu/127.12.34.254@KRBTEST.COM
I20260926 15:11:55.493036 25272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:11:55.474864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.195:56007 (local address 127.12.34.254:45857)
0926 15:11:55.475395 (+ 531us) server_negotiation.cc:206] Beginning negotiation
0926 15:11:55.475402 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:11:55.476726 (+ 1324us) server_negotiation.cc:407] Connection header received
0926 15:11:55.478463 (+ 1737us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:11:55.478468 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:11:55.478564 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:11:55.478688 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:11:55.480067 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:11:55.480791 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:11:55.482332 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:11:55.482848 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:11:55.482979 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:11:55.486533 (+ 3554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:11:55.486538 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:11:55.486542 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:11:55.486584 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:11:55.490041 (+ 3457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:11:55.490977 (+ 936us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:11:55.490981 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:11:55.490988 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:11:55.491121 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:11:55.491542 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:11:55.491545 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:11:55.491547 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:11:55.491816 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:11:55.491923 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 15:11:55.492588 (+ 665us) server_negotiation.cc:299] Negotiation successful
0926 15:11:55.492795 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":267,"threads_started":1}
I20260926 15:11:55.494112 25273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:11:55.475411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:45857 (local address 127.12.34.195:56007)
0926 15:11:55.476290 (+ 879us) negotiation.cc:107] Waiting for socket to connect
0926 15:11:55.476336 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 15:11:55.478186 (+ 1850us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:11:55.478905 (+ 719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:11:55.478913 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:11:55.479293 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:11:55.479904 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:11:55.479920 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:11:55.480942 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:11:55.480957 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:11:55.482174 (+ 1217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:11:55.482192 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:11:55.483066 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:11:55.483075 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:11:55.483325 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:11:55.484119 (+ 794us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:11:55.484143 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:11:55.486273 (+ 2130us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:11:55.490204 (+ 3931us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:11:55.490213 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:11:55.490230 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:11:55.490843 (+ 613us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:11:55.491256 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:11:55.491260 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:11:55.491262 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:11:55.491437 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:11:55.491923 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:11:55.491929 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:11:55.492241 (+ 312us) client_negotiation.cc:769] Sending connection context
0926 15:11:55.492459 (+ 218us) client_negotiation.cc:240] Negotiation successful
0926 15:11:55.492887 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":646,"thread_start_us":333,"threads_started":1}
I20260926 15:11:55.495453 25270 heartbeater.cc:347] Connected to a master server at 127.12.34.254:45857
I20260926 15:11:55.495746 25270 heartbeater.cc:464] Registering TS with master...
I20260926 15:11:55.496402 25270 heartbeater.cc:511] Master 127.12.34.254:45857 requested a full tablet report, sending...
I20260926 15:11:55.497920 24780 ts_manager.cc:194] Registered new tserver with Master: 5933f4a56ce04a8f9cc6c35ec5e7e243 (127.12.34.195:33371)
I20260926 15:11:55.498680 24780 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.34.195@KRBTEST.COM'} at 127.12.34.195:56007
I20260926 15:11:55.502741 12427 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 15:11:55.504343 12427 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 24860
Sep 26 15:11:55 dist-test-slave-865v krb5kdc[24304](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435511, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.34.193@KRBTEST.COM
I20260926 15:11:55.521267 25275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:11:55.505175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37454 (local address 127.12.34.193:33971)
0926 15:11:55.505746 (+ 571us) server_negotiation.cc:206] Beginning negotiation
0926 15:11:55.505752 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:11:55.505775 (+ 23us) server_negotiation.cc:407] Connection header received
0926 15:11:55.505830 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:11:55.505835 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:11:55.506006 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:11:55.506127 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:11:55.507653 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:11:55.508713 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:11:55.510493 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:11:55.511062 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:11:55.511175 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:11:55.515686 (+ 4511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:11:55.515696 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:11:55.515710 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:11:55.515747 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:11:55.518526 (+ 2779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:11:55.519214 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:11:55.519222 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:11:55.519224 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:11:55.519366 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:11:55.519706 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:11:55.519709 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:11:55.519711 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:11:55.520052 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:11:55.520280 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0926 15:11:55.520642 (+ 362us) server_negotiation.cc:299] Negotiation successful
0926 15:11:55.520888 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":435,"thread_start_us":270,"threads_started":1}
I20260926 15:11:55.522456 24988 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:11:55.605619 12427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8uoJGv/build/asan/bin/kudu with pid 24860
I20260926 15:11:55.621214 12427 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 24999
Sep 26 15:11:55 dist-test-slave-865v krb5kdc[24304](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435511, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.34.194@KRBTEST.COM
I20260926 15:11:55.636999 25277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:11:55.622099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34920 (local address 127.12.34.194:34699)
0926 15:11:55.622653 (+ 554us) server_negotiation.cc:206] Beginning negotiation
0926 15:11:55.622660 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:11:55.622686 (+ 26us) server_negotiation.cc:407] Connection header received
0926 15:11:55.622762 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:11:55.622769 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:11:55.622969 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:11:55.623109 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:11:55.624319 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:11:55.625379 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:11:55.626760 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:11:55.627278 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:11:55.627371 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:11:55.631555 (+ 4184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:11:55.631565 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:11:55.631577 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:11:55.631611 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:11:55.634306 (+ 2695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:11:55.634952 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:11:55.634960 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:11:55.634962 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:11:55.635088 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:11:55.635455 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:11:55.635461 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:11:55.635464 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:11:55.635783 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:11:55.636078 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0926 15:11:55.636418 (+ 340us) server_negotiation.cc:299] Negotiation successful
0926 15:11:55.636665 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":308,"threads_started":1}
I20260926 15:11:55.638228 25131 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:11:55.721632 12427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8uoJGv/build/asan/bin/kudu with pid 24999
I20260926 15:11:55.738138 12427 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 25142
Sep 26 15:11:55 dist-test-slave-865v krb5kdc[24304](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435511, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.34.195@KRBTEST.COM
I20260926 15:11:55.754562 25279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:11:55.739020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40188 (local address 127.12.34.195:33371)
0926 15:11:55.739614 (+ 594us) server_negotiation.cc:206] Beginning negotiation
0926 15:11:55.739623 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 15:11:55.739652 (+ 29us) server_negotiation.cc:407] Connection header received
0926 15:11:55.739710 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:11:55.739717 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:11:55.739922 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:11:55.740089 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:11:55.741474 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:11:55.742504 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:11:55.743765 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:11:55.744326 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:11:55.744473 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:11:55.749012 (+ 4539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:11:55.749023 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:11:55.749037 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:11:55.749069 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:11:55.751861 (+ 2792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:11:55.752545 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:11:55.752551 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:11:55.752554 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:11:55.752686 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:11:55.753058 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:11:55.753062 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:11:55.753064 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:11:55.753476 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:11:55.753726 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0926 15:11:55.753979 (+ 253us) server_negotiation.cc:299] Negotiation successful
0926 15:11:55.754162 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":450,"thread_start_us":305,"threads_started":1}
I20260926 15:11:55.755824 25265 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:11:55.838284 12427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8uoJGv/build/asan/bin/kudu with pid 25142
I20260926 15:11:55.853825 12427 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 24748
I20260926 15:11:55.855273 24810 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:11:55.966959 12427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8uoJGv/build/asan/bin/kudu with pid 24748
I20260926 15:11:55.991118 12427 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8uoJGv/build/asan/bin/kudu
/tmp/dist-test-task8uoJGv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.34.254:45857
--webserver_interface=127.12.34.254
--webserver_port=36207
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.254.keytab
--principal=kudu/127.12.34.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.34.212:45919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.34.254:45857
--ranger_config_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.34.212:60253/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.12.34.212:60253/kms with env {KRB5CCNAME=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:11:56.305500 25282 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:11:56.305763 25282 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:11:56.305929 25282 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:11:56.313058 25282 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:11:56.313218 25282 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:11:56.313280 25282 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:11:56.313331 25282 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:11:56.313377 25282 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:11:56.323199 25282 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.34.212:45919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.12.34.212:60253/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.12.34.254:45857
--ranger_config_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.254.keytab
--principal=kudu/127.12.34.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.34.254:45857
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.34.254
--webserver_port=36207
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:11:56.324337 25282 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:11:56.325539 25282 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:11:56.336906 25290 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:11:56.336962 25288 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:11:56.337142 25287 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:11:56.338116 25282 server_base.cc:1034] running on GCE node
I20260926 15:11:56.338833 25282 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:11:56.340108 25282 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:11:56.341351 25282 hybrid_clock.cc:648] HybridClock initialized: now 1790435516341318 us; error 29 us; skew 500 ppm
I20260926 15:11:56.341696 25282 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:11:56 dist-test-slave-865v krb5kdc[24304](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435516, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:11:56.344632 25282 init.cc:377] Logged in from keytab as kudu/127.12.34.254@KRBTEST.COM (short username kudu)
I20260926 15:11:56.346153 25282 webserver.cc:460] Webserver started at http://127.12.34.254:36207/ using document root <none> and password file <none>
I20260926 15:11:56.346737 25282 fs_manager.cc:359] Metadata directory not provided
I20260926 15:11:56.346839 25282 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 15:11:56 dist-test-slave-865v krb5kdc[24304](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435516, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.254@KRBTEST.COM for HTTP/127.12.34.212@KRBTEST.COM
I20260926 15:11:56.402133 25282 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 15:11:56.405328 25298 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:11:56.406930 25282 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 15:11:56.407106 25282 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/minicluster-data/master-0/data,/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/minicluster-data/master-0/wal
uuid: "503ca1bbe012419898d9ac8c1fa73872"
format_stamp: "Formatted at 2026-09-26 15:11:52 on dist-test-slave-865v"
server_key: "87e47af19dc8c5f65fc38de7cc316758"
server_key_iv: "5efb5d478a82fda602c96bdf507d5000"
server_key_version: "kuduclusterkey@0"
I20260926 15:11:56.408025 25282 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:11:56.434401 25282 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:11:56.438169 25282 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:11:56.438400 25282 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:11:56.457361 25282 rpc_server.cc:307] RPC server started. Bound to: 127.12.34.254:45857
I20260926 15:11:56.457379 25349 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.34.254:45857 every 8 connection(s)
I20260926 15:11:56.458770 25282 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
I20260926 15:11:56.463104 12427 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8uoJGv/build/asan/bin/kudu as pid 25282
I20260926 15:11:56.464150 12427 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8uoJGv/build/asan/bin/kudu
/tmp/dist-test-task8uoJGv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.34.212:60253/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.34.193:33971
--local_ip_for_outbound_sockets=127.12.34.193
--tserver_master_addrs=127.12.34.254:45857
--webserver_port=38989
--webserver_interface=127.12.34.193
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.193.keytab
--principal=kudu/127.12.34.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.34.212:45919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.12.34.212:60253/kms with env {KRB5CCNAME=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 15:11:56.466461 25350 sys_catalog.cc:263] Verifying existing consensus state
I20260926 15:11:56.471671 25350 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 503ca1bbe012419898d9ac8c1fa73872: Bootstrap starting.
I20260926 15:11:56.490550 25350 log.cc:824] T 00000000000000000000000000000000 P 503ca1bbe012419898d9ac8c1fa73872: Log is configured to *not* fsync() on all Append() calls
I20260926 15:11:56.499150 25350 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 503ca1bbe012419898d9ac8c1fa73872: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 15:11:56.499675 25350 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 503ca1bbe012419898d9ac8c1fa73872: Bootstrap complete.
I20260926 15:11:56.508402 25350 raft_consensus.cc:348] T 00000000000000000000000000000000 P 503ca1bbe012419898d9ac8c1fa73872 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "503ca1bbe012419898d9ac8c1fa73872" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 45857 } }
I20260926 15:11:56.509300 25350 raft_consensus.cc:729] T 00000000000000000000000000000000 P 503ca1bbe012419898d9ac8c1fa73872 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 503ca1bbe012419898d9ac8c1fa73872, State: Initialized, Role: FOLLOWER
I20260926 15:11:56.509905 25350 consensus_queue.cc:260] T 00000000000000000000000000000000 P 503ca1bbe012419898d9ac8c1fa73872 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "503ca1bbe012419898d9ac8c1fa73872" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 45857 } }
I20260926 15:11:56.510129 25350 raft_consensus.cc:388] T 00000000000000000000000000000000 P 503ca1bbe012419898d9ac8c1fa73872 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 15:11:56.510272 25350 raft_consensus.cc:482] T 00000000000000000000000000000000 P 503ca1bbe012419898d9ac8c1fa73872 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 15:11:56.510452 25350 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 503ca1bbe012419898d9ac8c1fa73872 [term 1 FOLLOWER]: Advancing to term 2
I20260926 15:11:56.514516 25350 raft_consensus.cc:504] T 00000000000000000000000000000000 P 503ca1bbe012419898d9ac8c1fa73872 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "503ca1bbe012419898d9ac8c1fa73872" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 45857 } }
I20260926 15:11:56.514995 25350 leader_election.cc:302] T 00000000000000000000000000000000 P 503ca1bbe012419898d9ac8c1fa73872 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 503ca1bbe012419898d9ac8c1fa73872; no voters:
I20260926 15:11:56.515604 25350 leader_election.cc:288] T 00000000000000000000000000000000 P 503ca1bbe012419898d9ac8c1fa73872 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 15:11:56.515740 25354 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 503ca1bbe012419898d9ac8c1fa73872 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 15:11:56.517938 25350 sys_catalog.cc:564] T 00000000000000000000000000000000 P 503ca1bbe012419898d9ac8c1fa73872 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:11:56.521462 25354 raft_consensus.cc:686] T 00000000000000000000000000000000 P 503ca1bbe012419898d9ac8c1fa73872 [term 2 LEADER]: Becoming Leader. State: Replica: 503ca1bbe012419898d9ac8c1fa73872, State: Running, Role: LEADER
I20260926 15:11:56.522094 25354 consensus_queue.cc:237] T 00000000000000000000000000000000 P 503ca1bbe012419898d9ac8c1fa73872 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "503ca1bbe012419898d9ac8c1fa73872" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 45857 } }
I20260926 15:11:56.524968 25350 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260926 15:11:56.524958 25355 sys_catalog.cc:455] T 00000000000000000000000000000000 P 503ca1bbe012419898d9ac8c1fa73872 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "503ca1bbe012419898d9ac8c1fa73872" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "503ca1bbe012419898d9ac8c1fa73872" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 45857 } } }
I20260926 15:11:56.525382 25355 sys_catalog.cc:458] T 00000000000000000000000000000000 P 503ca1bbe012419898d9ac8c1fa73872 [sys.catalog]: This master's current role is: LEADER
I20260926 15:11:56.526835 25356 sys_catalog.cc:455] T 00000000000000000000000000000000 P 503ca1bbe012419898d9ac8c1fa73872 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 503ca1bbe012419898d9ac8c1fa73872. Latest consensus state: current_term: 2 leader_uuid: "503ca1bbe012419898d9ac8c1fa73872" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "503ca1bbe012419898d9ac8c1fa73872" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 45857 } } }
I20260926 15:11:56.527211 25356 sys_catalog.cc:458] T 00000000000000000000000000000000 P 503ca1bbe012419898d9ac8c1fa73872 [sys.catalog]: This master's current role is: LEADER
I20260926 15:11:56.545274 25369 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:11:56.552795 25369 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 15:11:56.554570 25369 catalog_manager.cc:1259] Loaded cluster ID: 5187116344214de2a22feb8eb50ee8aa
I20260926 15:11:56.554730 25369 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 15:11:56.559579 25369 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:11:56.561682 25369 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 503ca1bbe012419898d9ac8c1fa73872: Loaded TSK: 0
I20260926 15:11:56.563504 25369 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 15:11:56.914637 25352 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:11:56.915076 25352 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:11:56.915248 25352 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:11:56.927723 25352 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:11:56.927961 25352 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:11:56.928215 25352 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.34.193
I20260926 15:11:56.944409 25352 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.34.212:45919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.12.34.212:60253/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.193.keytab
--principal=kudu/127.12.34.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.34.193:33971
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.34.193
--webserver_port=38989
--tserver_master_addrs=127.12.34.254:45857
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.34.193
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:11:56.946473 25352 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:11:56.948328 25352 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:11:56.980329 25387 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:11:56.980353 25390 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:11:56.980289 25388 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:11:56.982141 25352 server_base.cc:1034] running on GCE node
I20260926 15:11:56.982966 25352 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:11:56.984436 25352 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:11:56.985777 25352 hybrid_clock.cc:648] HybridClock initialized: now 1790435516985700 us; error 57 us; skew 500 ppm
I20260926 15:11:56.986234 25352 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:11:56 dist-test-slave-865v krb5kdc[24304](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435516, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:11:56.995795 25352 init.cc:377] Logged in from keytab as kudu/127.12.34.193@KRBTEST.COM (short username kudu)
I20260926 15:11:57.006747 25352 webserver.cc:460] Webserver started at http://127.12.34.193:38989/ using document root <none> and password file <none>
I20260926 15:11:57.007812 25352 fs_manager.cc:359] Metadata directory not provided
I20260926 15:11:57.008008 25352 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 15:11:57 dist-test-slave-865v krb5kdc[24304](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435516, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.193@KRBTEST.COM for HTTP/127.12.34.212@KRBTEST.COM
I20260926 15:11:57.061444 25352 fs_manager.cc:709] Time spent opening directory manager: real 0.026s user 0.008s sys 0.011s
I20260926 15:11:57.065763 25398 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:11:57.067780 25352 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260926 15:11:57.067993 25352 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/minicluster-data/ts-0/data,/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/minicluster-data/ts-0/wal
uuid: "a26ca453e2c941d2b4d5e9972af91fde"
format_stamp: "Formatted at 2026-09-26 15:11:53 on dist-test-slave-865v"
server_key: "daf77363c027870de5bd2476911c363f"
server_key_iv: "6dfe77c37baca6ef60e01abf9aa1759a"
server_key_version: "kuduclusterkey@0"
I20260926 15:11:57.069579 25352 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:11:57.093235 25352 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:11:57.098690 25352 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:11:57.098996 25352 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:11:57.102881 25352 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:11:57.105922 25352 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:11:57.106110 25352 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260926 15:11:57.106235 25352 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:11:57.106329 25352 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:11:57.174124 25352 rpc_server.cc:307] RPC server started. Bound to: 127.12.34.193:33971
I20260926 15:11:57.174171 25511 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.34.193:33971 every 8 connection(s)
I20260926 15:11:57.175673 25352 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/minicluster-data/ts-0/data/info.pb
I20260926 15:11:57.179046 12427 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8uoJGv/build/asan/bin/kudu as pid 25352
I20260926 15:11:57.180377 12427 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8uoJGv/build/asan/bin/kudu
/tmp/dist-test-task8uoJGv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.34.212:60253/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.34.194:34699
--local_ip_for_outbound_sockets=127.12.34.194
--tserver_master_addrs=127.12.34.254:45857
--webserver_port=38227
--webserver_interface=127.12.34.194
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.194.keytab
--principal=kudu/127.12.34.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.34.212:45919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.12.34.212:60253/kms with env {KRB5CCNAME=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 15:11:57 dist-test-slave-865v krb5kdc[24304](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435516, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.193@KRBTEST.COM for kudu/127.12.34.254@KRBTEST.COM
I20260926 15:11:57.202571 25515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:11:57.179068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:45857 (local address 127.12.34.193:34315)
0926 15:11:57.180235 (+ 1167us) negotiation.cc:107] Waiting for socket to connect
0926 15:11:57.180288 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 15:11:57.182581 (+ 2293us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:11:57.184463 (+ 1882us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:11:57.184476 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:11:57.185209 (+ 733us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:11:57.185952 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:11:57.185991 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:11:57.188237 (+ 2246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:11:57.188242 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:11:57.189562 (+ 1320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:11:57.189574 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:11:57.190649 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:11:57.190656 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:11:57.190902 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:11:57.191591 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:11:57.191644 (+ 53us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:11:57.193868 (+ 2224us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:11:57.197696 (+ 3828us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:11:57.197705 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:11:57.197721 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:11:57.198272 (+ 551us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:11:57.198818 (+ 546us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:11:57.198823 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:11:57.198827 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:11:57.199044 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:11:57.200003 (+ 959us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:11:57.200011 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:11:57.200384 (+ 373us) client_negotiation.cc:769] Sending connection context
0926 15:11:57.200678 (+ 294us) client_negotiation.cc:240] Negotiation successful
0926 15:11:57.201228 (+ 550us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":861,"thread_start_us":641,"threads_started":1}
I20260926 15:11:57.205495 25514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:11:57.178955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.193:34315 (local address 127.12.34.254:45857)
0926 15:11:57.183072 (+ 4117us) server_negotiation.cc:206] Beginning negotiation
0926 15:11:57.183089 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 15:11:57.183126 (+ 37us) server_negotiation.cc:407] Connection header received
0926 15:11:57.183494 (+ 368us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:11:57.183510 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:11:57.183922 (+ 412us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:11:57.184278 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:11:57.186204 (+ 1926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:11:57.188087 (+ 1883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:11:57.189718 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:11:57.190398 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:11:57.190607 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:11:57.194160 (+ 3553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:11:57.194168 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:11:57.194183 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:11:57.194216 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:11:57.197521 (+ 3305us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:11:57.198446 (+ 925us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:11:57.198455 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:11:57.198463 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:11:57.198672 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:11:57.199182 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:11:57.199187 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:11:57.199191 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:11:57.199841 (+ 650us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:11:57.203391 (+ 3550us) server_negotiation.cc:1035] Waiting for connection context
0926 15:11:57.203645 (+ 254us) server_negotiation.cc:299] Negotiation successful
0926 15:11:57.204174 (+ 529us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":613,"thread_start_us":400,"threads_started":1}
I20260926 15:11:57.208747 25512 heartbeater.cc:347] Connected to a master server at 127.12.34.254:45857
I20260926 15:11:57.209133 25512 heartbeater.cc:464] Registering TS with master...
I20260926 15:11:57.209903 25512 heartbeater.cc:511] Master 127.12.34.254:45857 requested a full tablet report, sending...
W20260926 15:11:57.212932 25512 heartbeater.cc:644] Failed to heartbeat to 127.12.34.254:45857 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 15:11:57.213199 25512 heartbeater.cc:464] Registering TS with master...
I20260926 15:11:57.213483 25512 heartbeater.cc:511] Master 127.12.34.254:45857 requested a full tablet report, sending...
I20260926 15:11:57.214614 25512 heartbeater.cc:464] Registering TS with master...
I20260926 15:11:57.214978 25512 heartbeater.cc:511] Master 127.12.34.254:45857 requested a full tablet report, sending...
W20260926 15:11:57.216212 25512 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 15:11:57.592687 25516 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:11:57.592967 25516 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:11:57.593182 25516 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:11:57.601586 25516 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:11:57.601725 25516 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:11:57.601897 25516 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.34.194
I20260926 15:11:57.611660 25516 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.34.212:45919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.12.34.212:60253/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.194.keytab
--principal=kudu/127.12.34.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.34.194:34699
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.34.194
--webserver_port=38227
--tserver_master_addrs=127.12.34.254:45857
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.34.194
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:11:57.613173 25516 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:11:57.614790 25516 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:11:57.635032 25530 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:11:57.637218 25531 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:11:57.637332 25533 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:11:57.637472 25516 server_base.cc:1034] running on GCE node
I20260926 15:11:57.638329 25516 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:11:57.639339 25516 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:11:57.640596 25516 hybrid_clock.cc:648] HybridClock initialized: now 1790435517640532 us; error 52 us; skew 500 ppm
I20260926 15:11:57.640969 25516 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:11:57 dist-test-slave-865v krb5kdc[24304](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435517, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:11:57.644919 25516 init.cc:377] Logged in from keytab as kudu/127.12.34.194@KRBTEST.COM (short username kudu)
I20260926 15:11:57.650556 25516 webserver.cc:460] Webserver started at http://127.12.34.194:38227/ using document root <none> and password file <none>
I20260926 15:11:57.651582 25516 fs_manager.cc:359] Metadata directory not provided
I20260926 15:11:57.651774 25516 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 15:11:57 dist-test-slave-865v krb5kdc[24304](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435517, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.194@KRBTEST.COM for HTTP/127.12.34.212@KRBTEST.COM
I20260926 15:11:57.729143 25516 fs_manager.cc:709] Time spent opening directory manager: real 0.015s user 0.007s sys 0.003s
I20260926 15:11:57.745764 25543 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:11:57.747890 25516 fs_manager.cc:725] Time spent opening block manager: real 0.017s user 0.010s sys 0.006s
I20260926 15:11:57.748186 25516 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/minicluster-data/ts-1/data,/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/minicluster-data/ts-1/wal
uuid: "ed30be0b68124b0ead8c15e6ffe0aef7"
format_stamp: "Formatted at 2026-09-26 15:11:54 on dist-test-slave-865v"
server_key: "c29c29b048fec2032595a9e88c75e89f"
server_key_iv: "d8e1631c29166075a8f42f365e931179"
server_key_version: "kuduclusterkey@0"
I20260926 15:11:57.749831 25516 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:11:57.775516 25516 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
Sep 26 15:11:57 dist-test-slave-865v krb5kdc[24304](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790435517, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:11:57.781286 25516 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:11:57.781672 25516 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:11:57.789256 25516 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:11:57.797499 25516 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:11:57.797691 25516 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:11:57.797859 25516 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:11:57.797986 25516 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:11:57.855103 25516 rpc_server.cc:307] RPC server started. Bound to: 127.12.34.194:34699
I20260926 15:11:57.855146 25655 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.34.194:34699 every 8 connection(s)
I20260926 15:11:57.856953 25516 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/minicluster-data/ts-1/data/info.pb
I20260926 15:11:57.859588 12427 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8uoJGv/build/asan/bin/kudu as pid 25516
I20260926 15:11:57.860337 12427 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8uoJGv/build/asan/bin/kudu
/tmp/dist-test-task8uoJGv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.34.212:60253/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.34.195:33371
--local_ip_for_outbound_sockets=127.12.34.195
--tserver_master_addrs=127.12.34.254:45857
--webserver_port=41741
--webserver_interface=127.12.34.195
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.195.keytab
--principal=kudu/127.12.34.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.34.212:45919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.12.34.212:60253/kms with env {KRB5CCNAME=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 15:11:57 dist-test-slave-865v krb5kdc[24304](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435517, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.194@KRBTEST.COM for kudu/127.12.34.254@KRBTEST.COM
I20260926 15:11:57.897886 25658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:11:57.859824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.194:52651 (local address 127.12.34.254:45857)
0926 15:11:57.876487 (+ 16663us) server_negotiation.cc:206] Beginning negotiation
0926 15:11:57.876491 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:11:57.876510 (+ 19us) server_negotiation.cc:407] Connection header received
0926 15:11:57.876563 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:11:57.876566 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:11:57.876629 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:11:57.876721 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:11:57.879008 (+ 2287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:11:57.880261 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:11:57.882904 (+ 2643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:11:57.883676 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:11:57.883812 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:11:57.888835 (+ 5023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:11:57.888845 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:11:57.888853 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:11:57.888939 (+ 86us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:11:57.892986 (+ 4047us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:11:57.894086 (+ 1100us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:11:57.894095 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:11:57.894102 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:11:57.894257 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:11:57.894835 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:11:57.894843 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:11:57.894848 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:11:57.895254 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:11:57.895413 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0926 15:11:57.897300 (+ 1887us) server_negotiation.cc:299] Negotiation successful
0926 15:11:57.897525 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16556,"thread_start_us":16446,"threads_started":1}
I20260926 15:11:57.899385 25659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:11:57.860343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:45857 (local address 127.12.34.194:52651)
0926 15:11:57.872260 (+ 11917us) negotiation.cc:107] Waiting for socket to connect
0926 15:11:57.872319 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0926 15:11:57.874936 (+ 2617us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:11:57.877152 (+ 2216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:11:57.877165 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:11:57.877872 (+ 707us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:11:57.878771 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:11:57.878804 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:11:57.880448 (+ 1644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:11:57.880456 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:11:57.882681 (+ 2225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:11:57.882706 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:11:57.884107 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:11:57.884117 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:11:57.884428 (+ 311us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:11:57.885477 (+ 1049us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:11:57.885523 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:11:57.888536 (+ 3013us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:11:57.893176 (+ 4640us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:11:57.893188 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:11:57.893210 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:11:57.893902 (+ 692us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:11:57.894402 (+ 500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:11:57.894407 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:11:57.894412 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:11:57.894683 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:11:57.895492 (+ 809us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:11:57.895503 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:11:57.896007 (+ 504us) client_negotiation.cc:769] Sending connection context
0926 15:11:57.896336 (+ 329us) client_negotiation.cc:240] Negotiation successful
0926 15:11:57.896946 (+ 610us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11604,"thread_start_us":11353,"threads_started":1}
I20260926 15:11:57.901410 25656 heartbeater.cc:347] Connected to a master server at 127.12.34.254:45857
I20260926 15:11:57.901713 25656 heartbeater.cc:464] Registering TS with master...
I20260926 15:11:57.902522 25656 heartbeater.cc:511] Master 127.12.34.254:45857 requested a full tablet report, sending...
I20260926 15:11:57.904488 25315 ts_manager.cc:194] Registered new tserver with Master: ed30be0b68124b0ead8c15e6ffe0aef7 (127.12.34.194:34699)
I20260926 15:11:57.906597 25315 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.34.194@KRBTEST.COM'} at 127.12.34.194:52651
I20260926 15:11:58.216372 25512 heartbeater.cc:464] Registering TS with master...
I20260926 15:11:58.216828 25512 heartbeater.cc:511] Master 127.12.34.254:45857 requested a full tablet report, sending...
I20260926 15:11:58.218343 25315 ts_manager.cc:194] Registered new tserver with Master: a26ca453e2c941d2b4d5e9972af91fde (127.12.34.193:33971)
I20260926 15:11:58.219501 25315 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.34.193@KRBTEST.COM'} at 127.12.34.193:34315
W20260926 15:11:58.263870 25660 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:11:58.264218 25660 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:11:58.264355 25660 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:11:58.274406 25660 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:11:58.274530 25660 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:11:58.274695 25660 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.34.195
Sep 26 15:11:58 dist-test-slave-865v krb5kdc[24304](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435517, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.254@KRBTEST.COM for HTTP/127.12.34.212@KRBTEST.COM
I20260926 15:11:58.289105 25660 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.34.212:45919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.12.34.212:60253/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.195.keytab
--principal=kudu/127.12.34.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.34.195:33371
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.34.195
--webserver_port=41741
--tserver_master_addrs=127.12.34.254:45857
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.34.195
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:11:58.290737 25660 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:11:58.292567 25660 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:11:58.313134 25667 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:11:58.314072 25666 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:11:58.317991 25669 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:11:58.319388 25660 server_base.cc:1034] running on GCE node
I20260926 15:11:58.320225 25660 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:11:58.330523 25660 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:11:58.333988 25660 hybrid_clock.cc:648] HybridClock initialized: now 1790435518333917 us; error 69 us; skew 500 ppm
I20260926 15:11:58.334396 25660 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:11:58 dist-test-slave-865v krb5kdc[24304](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435518, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:11:58.338332 25660 init.cc:377] Logged in from keytab as kudu/127.12.34.195@KRBTEST.COM (short username kudu)
I20260926 15:11:58.357256 25660 webserver.cc:460] Webserver started at http://127.12.34.195:41741/ using document root <none> and password file <none>
I20260926 15:11:58.358299 25660 fs_manager.cc:359] Metadata directory not provided
I20260926 15:11:58.358479 25660 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 15:11:58 dist-test-slave-865v krb5kdc[24304](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435518, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.195@KRBTEST.COM for HTTP/127.12.34.212@KRBTEST.COM
I20260926 15:11:58.390249 25660 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.002s sys 0.001s
I20260926 15:11:58.393648 25677 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:11:58.395316 25660 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260926 15:11:58.395529 25660 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/minicluster-data/ts-2/data,/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/minicluster-data/ts-2/wal
uuid: "5933f4a56ce04a8f9cc6c35ec5e7e243"
format_stamp: "Formatted at 2026-09-26 15:11:55 on dist-test-slave-865v"
server_key: "f784631f0d2979c1cac701e8716732a8"
server_key_iv: "65eb352b56df397fc0e739d8d7c341cc"
server_key_version: "kuduclusterkey@0"
I20260926 15:11:58.396872 25660 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:11:58.434310 25660 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:11:58.438308 25660 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:11:58.438592 25660 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:11:58.439667 25660 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:11:58.441197 25660 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:11:58.441304 25660 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:11:58.441427 25660 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:11:58.441519 25660 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:11:58.497742 25660 rpc_server.cc:307] RPC server started. Bound to: 127.12.34.195:33371
I20260926 15:11:58.497779 25789 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.34.195:33371 every 8 connection(s)
I20260926 15:11:58.499788 25660 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/minicluster-data/ts-2/data/info.pb
I20260926 15:11:58.501458 12427 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8uoJGv/build/asan/bin/kudu as pid 25660
Sep 26 15:11:58 dist-test-slave-865v krb5kdc[24304](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435518, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.195@KRBTEST.COM for kudu/127.12.34.254@KRBTEST.COM
I20260926 15:11:58.538582 25793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:11:58.503539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:45857 (local address 127.12.34.195:47743)
0926 15:11:58.513210 (+ 9671us) negotiation.cc:107] Waiting for socket to connect
0926 15:11:58.513267 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0926 15:11:58.515612 (+ 2345us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:11:58.517489 (+ 1877us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:11:58.517507 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:11:58.518384 (+ 877us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:11:58.519531 (+ 1147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:11:58.519559 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:11:58.521232 (+ 1673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:11:58.521238 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:11:58.523419 (+ 2181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:11:58.523435 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:11:58.524841 (+ 1406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:11:58.524848 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:11:58.525172 (+ 324us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:11:58.526087 (+ 915us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:11:58.526124 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:11:58.528966 (+ 2842us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:11:58.533597 (+ 4631us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:11:58.533607 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:11:58.533625 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:11:58.534259 (+ 634us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:11:58.534707 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:11:58.534712 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:11:58.534716 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:11:58.534966 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:11:58.535746 (+ 780us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:11:58.535756 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:11:58.536210 (+ 454us) client_negotiation.cc:769] Sending connection context
0926 15:11:58.536535 (+ 325us) client_negotiation.cc:240] Negotiation successful
0926 15:11:58.537145 (+ 610us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9333,"thread_start_us":13478,"threads_started":1}
I20260926 15:11:58.538837 25794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:11:58.503854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43510 (local address 127.12.34.254:45857)
0926 15:11:58.516339 (+ 12485us) server_negotiation.cc:206] Beginning negotiation
0926 15:11:58.516346 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:11:58.518515 (+ 2169us) server_negotiation.cc:407] Connection header received
0926 15:11:58.518766 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:11:58.518772 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:11:58.518852 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:11:58.518965 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:11:58.521149 (+ 2184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:11:58.522383 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:11:58.524351 (+ 1968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:11:58.525035 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:11:58.525153 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:11:58.528640 (+ 3487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:11:58.528646 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:11:58.528651 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:11:58.528693 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:11:58.533751 (+ 5058us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:11:58.535971 (+ 2220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:11:58.535977 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:11:58.535980 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:11:58.536124 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:11:58.536562 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:11:58.536567 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:11:58.536570 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:11:58.536866 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:11:58.537124 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0926 15:11:58.538259 (+ 1135us) server_negotiation.cc:299] Negotiation successful
0926 15:11:58.538488 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12354,"thread_start_us":12241,"threads_started":1}
I20260926 15:11:58.540611 25795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:11:58.504188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.195:47743 (local address 127.12.34.254:45857)
0926 15:11:58.516664 (+ 12476us) server_negotiation.cc:206] Beginning negotiation
0926 15:11:58.516670 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:11:58.516688 (+ 18us) server_negotiation.cc:407] Connection header received
0926 15:11:58.516761 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:11:58.516766 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:11:58.516856 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:11:58.516990 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:11:58.519770 (+ 2780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:11:58.520987 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:11:58.523625 (+ 2638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:11:58.524358 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:11:58.524496 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:11:58.529308 (+ 4812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:11:58.529315 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:11:58.529319 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:11:58.529375 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:11:58.533422 (+ 4047us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:11:58.534414 (+ 992us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:11:58.534419 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:11:58.534423 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:11:58.534566 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:11:58.535114 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:11:58.535120 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:11:58.535123 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:11:58.535485 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:11:58.535667 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0926 15:11:58.539966 (+ 4299us) server_negotiation.cc:299] Negotiation successful
0926 15:11:58.540189 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12375,"thread_start_us":12468,"threads_started":1}
I20260926 15:11:58.542454 25790 heartbeater.cc:347] Connected to a master server at 127.12.34.254:45857
I20260926 15:11:58.542796 25790 heartbeater.cc:464] Registering TS with master...
I20260926 15:11:58.543511 25790 heartbeater.cc:511] Master 127.12.34.254:45857 requested a full tablet report, sending...
I20260926 15:11:58.544894 25315 ts_manager.cc:194] Registered new tserver with Master: 5933f4a56ce04a8f9cc6c35ec5e7e243 (127.12.34.195:33371)
I20260926 15:11:58.545769 25315 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.34.195@KRBTEST.COM'} at 127.12.34.195:47743
I20260926 15:11:58.553925 12427 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 15:11:58.578519 25795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:11:58.557281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43516 (local address 127.12.34.254:45857)
0926 15:11:58.557468 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0926 15:11:58.557477 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 15:11:58.557970 (+ 493us) server_negotiation.cc:407] Connection header received
0926 15:11:58.558163 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:11:58.558168 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:11:58.558240 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:11:58.558336 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:11:58.559748 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:11:58.560456 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:11:58.562383 (+ 1927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:11:58.562811 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:11:58.562882 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:11:58.567732 (+ 4850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:11:58.567740 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:11:58.567745 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:11:58.567810 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:11:58.574421 (+ 6611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:11:58.575368 (+ 947us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:11:58.575375 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:11:58.575400 (+ 25us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:11:58.575568 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:11:58.576100 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:11:58.576105 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:11:58.576109 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:11:58.576462 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:11:58.576701 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0926 15:11:58.577958 (+ 1257us) server_negotiation.cc:299] Negotiation successful
0926 15:11:58.578180 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 15:11:58.584131 25315 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43516:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 15:11:58.595219 25315 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
Sep 26 15:11:58 dist-test-slave-865v krb5kdc[24304](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435517, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.254@KRBTEST.COM for HTTP/127.12.34.212@KRBTEST.COM
I20260926 15:11:58.671694 25807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:11:58.650660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34934 (local address 127.12.34.194:34699)
0926 15:11:58.651256 (+ 596us) server_negotiation.cc:206] Beginning negotiation
0926 15:11:58.651267 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 15:11:58.662342 (+ 11075us) server_negotiation.cc:407] Connection header received
0926 15:11:58.662744 (+ 402us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:11:58.662754 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:11:58.663003 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:11:58.663173 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:11:58.664126 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:11:58.665730 (+ 1604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:11:58.668687 (+ 2957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:11:58.670188 (+ 1501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:11:58.670287 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:11:58.670490 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0926 15:11:58.671153 (+ 663us) server_negotiation.cc:299] Negotiation successful
0926 15:11:58.671333 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":606,"threads_started":1}
I20260926 15:11:58.672514 25808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:11:58.650106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.194:34699 (local address 127.0.0.1:34934)
0926 15:11:58.662192 (+ 12086us) negotiation.cc:107] Waiting for socket to connect
0926 15:11:58.662228 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 15:11:58.662493 (+ 265us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:11:58.663493 (+ 1000us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:11:58.663500 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:11:58.663531 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:11:58.663921 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:11:58.663933 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:11:58.665913 (+ 1980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:11:58.665918 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:11:58.668494 (+ 2576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:11:58.668509 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:11:58.670680 (+ 2171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:11:58.670686 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:11:58.670879 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:11:58.670929 (+ 50us) client_negotiation.cc:769] Sending connection context
0926 15:11:58.671913 (+ 984us) client_negotiation.cc:240] Negotiation successful
0926 15:11:58.672146 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11834,"threads_started":1}
I20260926 15:11:58.678158 25591 tablet_service.cc:1467] Processing CreateTablet for tablet e4c7e54e5d2c449dbe4a638a69fc6f30 (DEFAULT_TABLE table=test-table [id=4c55bb836ccd49b39cdebae8fa8cc563]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:11:58.680227 25591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4c7e54e5d2c449dbe4a638a69fc6f30. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:11:58.682849 25809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:11:58.650822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.195:33371 (local address 127.0.0.1:40192)
0926 15:11:58.666605 (+ 15783us) negotiation.cc:107] Waiting for socket to connect
0926 15:11:58.666623 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 15:11:58.666753 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:11:58.668180 (+ 1427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:11:58.668184 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:11:58.668208 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:11:58.668596 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:11:58.668607 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:11:58.672024 (+ 3417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:11:58.672029 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:11:58.673714 (+ 1685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:11:58.673728 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:11:58.681578 (+ 7850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:11:58.681585 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:11:58.682168 (+ 583us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:11:58.682206 (+ 38us) client_negotiation.cc:769] Sending connection context
0926 15:11:58.682318 (+ 112us) client_negotiation.cc:240] Negotiation successful
0926 15:11:58.682462 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15630,"thread_start_us":10969,"threads_started":1}
I20260926 15:11:58.683796 25811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:11:58.651484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40192 (local address 127.12.34.195:33371)
0926 15:11:58.661585 (+ 10101us) server_negotiation.cc:206] Beginning negotiation
0926 15:11:58.661596 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 15:11:58.666900 (+ 5304us) server_negotiation.cc:407] Connection header received
0926 15:11:58.667000 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:11:58.667007 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:11:58.667238 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:11:58.667410 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:11:58.668801 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:11:58.670400 (+ 1599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:11:58.673899 (+ 3499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:11:58.681047 (+ 7148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:11:58.681196 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:11:58.681413 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0926 15:11:58.683266 (+ 1853us) server_negotiation.cc:299] Negotiation successful
0926 15:11:58.683438 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9899,"thread_start_us":9768,"threads_started":1}
I20260926 15:11:58.690855 25725 tablet_service.cc:1467] Processing CreateTablet for tablet e4c7e54e5d2c449dbe4a638a69fc6f30 (DEFAULT_TABLE table=test-table [id=4c55bb836ccd49b39cdebae8fa8cc563]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:11:58.692303 25725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4c7e54e5d2c449dbe4a638a69fc6f30. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:11:58.702153 25810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:11:58.651385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37466 (local address 127.12.34.193:33971)
0926 15:11:58.652048 (+ 663us) server_negotiation.cc:206] Beginning negotiation
0926 15:11:58.652069 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0926 15:11:58.668112 (+ 16043us) server_negotiation.cc:407] Connection header received
0926 15:11:58.670816 (+ 2704us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:11:58.670825 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:11:58.671054 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:11:58.671214 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:11:58.676422 (+ 5208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:11:58.677894 (+ 1472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:11:58.698781 (+ 20887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:11:58.700306 (+ 1525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:11:58.700426 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:11:58.700653 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0926 15:11:58.701547 (+ 894us) server_negotiation.cc:299] Negotiation successful
0926 15:11:58.701736 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":478,"thread_start_us":302,"threads_started":1}
I20260926 15:11:58.702914 25806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:11:58.651126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.193:33971 (local address 127.0.0.1:37466)
0926 15:11:58.667992 (+ 16866us) negotiation.cc:107] Waiting for socket to connect
0926 15:11:58.668013 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 15:11:58.670596 (+ 2583us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:11:58.671496 (+ 900us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:11:58.671502 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:11:58.671525 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:11:58.671855 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:11:58.671866 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:11:58.678128 (+ 6262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:11:58.678133 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:11:58.680098 (+ 1965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:11:58.680114 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:11:58.700996 (+ 20882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:11:58.701005 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:11:58.701209 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:11:58.701362 (+ 153us) client_negotiation.cc:769] Sending connection context
0926 15:11:58.702450 (+ 1088us) client_negotiation.cc:240] Negotiation successful
0926 15:11:58.702613 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16757,"thread_start_us":15521,"threads_started":1}
I20260926 15:11:58.706750 25447 tablet_service.cc:1467] Processing CreateTablet for tablet e4c7e54e5d2c449dbe4a638a69fc6f30 (DEFAULT_TABLE table=test-table [id=4c55bb836ccd49b39cdebae8fa8cc563]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:11:58.708195 25447 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4c7e54e5d2c449dbe4a638a69fc6f30. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:11:58.710776 25812 tablet_bootstrap.cc:492] T e4c7e54e5d2c449dbe4a638a69fc6f30 P ed30be0b68124b0ead8c15e6ffe0aef7: Bootstrap starting.
I20260926 15:11:58.714504 25812 tablet_bootstrap.cc:654] T e4c7e54e5d2c449dbe4a638a69fc6f30 P ed30be0b68124b0ead8c15e6ffe0aef7: Neither blocks nor log segments found. Creating new log.
I20260926 15:11:58.715811 25812 log.cc:824] T e4c7e54e5d2c449dbe4a638a69fc6f30 P ed30be0b68124b0ead8c15e6ffe0aef7: Log is configured to *not* fsync() on all Append() calls
I20260926 15:11:58.727870 25812 tablet_bootstrap.cc:492] T e4c7e54e5d2c449dbe4a638a69fc6f30 P ed30be0b68124b0ead8c15e6ffe0aef7: No bootstrap required, opened a new log
I20260926 15:11:58.728181 25812 ts_tablet_manager.cc:1397] T e4c7e54e5d2c449dbe4a638a69fc6f30 P ed30be0b68124b0ead8c15e6ffe0aef7: Time spent bootstrapping tablet: real 0.018s user 0.006s sys 0.000s
I20260926 15:11:58.737783 25814 tablet_bootstrap.cc:492] T e4c7e54e5d2c449dbe4a638a69fc6f30 P a26ca453e2c941d2b4d5e9972af91fde: Bootstrap starting.
I20260926 15:11:58.741232 25815 tablet_bootstrap.cc:492] T e4c7e54e5d2c449dbe4a638a69fc6f30 P 5933f4a56ce04a8f9cc6c35ec5e7e243: Bootstrap starting.
I20260926 15:11:58.743652 25814 tablet_bootstrap.cc:654] T e4c7e54e5d2c449dbe4a638a69fc6f30 P a26ca453e2c941d2b4d5e9972af91fde: Neither blocks nor log segments found. Creating new log.
I20260926 15:11:58.744710 25815 tablet_bootstrap.cc:654] T e4c7e54e5d2c449dbe4a638a69fc6f30 P 5933f4a56ce04a8f9cc6c35ec5e7e243: Neither blocks nor log segments found. Creating new log.
I20260926 15:11:58.744978 25814 log.cc:824] T e4c7e54e5d2c449dbe4a638a69fc6f30 P a26ca453e2c941d2b4d5e9972af91fde: Log is configured to *not* fsync() on all Append() calls
I20260926 15:11:58.746027 25815 log.cc:824] T e4c7e54e5d2c449dbe4a638a69fc6f30 P 5933f4a56ce04a8f9cc6c35ec5e7e243: Log is configured to *not* fsync() on all Append() calls
I20260926 15:11:58.747083 25812 raft_consensus.cc:348] T e4c7e54e5d2c449dbe4a638a69fc6f30 P ed30be0b68124b0ead8c15e6ffe0aef7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed30be0b68124b0ead8c15e6ffe0aef7" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 34699 } } peers { permanent_uuid: "a26ca453e2c941d2b4d5e9972af91fde" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 33971 } } peers { permanent_uuid: "5933f4a56ce04a8f9cc6c35ec5e7e243" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 33371 } }
I20260926 15:11:58.747571 25812 raft_consensus.cc:374] T e4c7e54e5d2c449dbe4a638a69fc6f30 P ed30be0b68124b0ead8c15e6ffe0aef7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:11:58.747677 25812 raft_consensus.cc:729] T e4c7e54e5d2c449dbe4a638a69fc6f30 P ed30be0b68124b0ead8c15e6ffe0aef7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed30be0b68124b0ead8c15e6ffe0aef7, State: Initialized, Role: FOLLOWER
I20260926 15:11:58.754285 25812 consensus_queue.cc:260] T e4c7e54e5d2c449dbe4a638a69fc6f30 P ed30be0b68124b0ead8c15e6ffe0aef7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed30be0b68124b0ead8c15e6ffe0aef7" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 34699 } } peers { permanent_uuid: "a26ca453e2c941d2b4d5e9972af91fde" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 33971 } } peers { permanent_uuid: "5933f4a56ce04a8f9cc6c35ec5e7e243" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 33371 } }
I20260926 15:11:58.754846 25814 tablet_bootstrap.cc:492] T e4c7e54e5d2c449dbe4a638a69fc6f30 P a26ca453e2c941d2b4d5e9972af91fde: No bootstrap required, opened a new log
I20260926 15:11:58.755012 25815 tablet_bootstrap.cc:492] T e4c7e54e5d2c449dbe4a638a69fc6f30 P 5933f4a56ce04a8f9cc6c35ec5e7e243: No bootstrap required, opened a new log
I20260926 15:11:58.755146 25814 ts_tablet_manager.cc:1397] T e4c7e54e5d2c449dbe4a638a69fc6f30 P a26ca453e2c941d2b4d5e9972af91fde: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.010s
I20260926 15:11:58.756770 25815 ts_tablet_manager.cc:1397] T e4c7e54e5d2c449dbe4a638a69fc6f30 P 5933f4a56ce04a8f9cc6c35ec5e7e243: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.009s
I20260926 15:11:58.765882 25656 heartbeater.cc:503] Master 127.12.34.254:45857 was elected leader, sending a full tablet report...
I20260926 15:11:58.766358 25814 raft_consensus.cc:348] T e4c7e54e5d2c449dbe4a638a69fc6f30 P a26ca453e2c941d2b4d5e9972af91fde [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed30be0b68124b0ead8c15e6ffe0aef7" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 34699 } } peers { permanent_uuid: "a26ca453e2c941d2b4d5e9972af91fde" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 33971 } } peers { permanent_uuid: "5933f4a56ce04a8f9cc6c35ec5e7e243" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 33371 } }
I20260926 15:11:58.766824 25814 raft_consensus.cc:374] T e4c7e54e5d2c449dbe4a638a69fc6f30 P a26ca453e2c941d2b4d5e9972af91fde [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:11:58.766954 25814 raft_consensus.cc:729] T e4c7e54e5d2c449dbe4a638a69fc6f30 P a26ca453e2c941d2b4d5e9972af91fde [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a26ca453e2c941d2b4d5e9972af91fde, State: Initialized, Role: FOLLOWER
I20260926 15:11:58.767709 25814 consensus_queue.cc:260] T e4c7e54e5d2c449dbe4a638a69fc6f30 P a26ca453e2c941d2b4d5e9972af91fde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed30be0b68124b0ead8c15e6ffe0aef7" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 34699 } } peers { permanent_uuid: "a26ca453e2c941d2b4d5e9972af91fde" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 33971 } } peers { permanent_uuid: "5933f4a56ce04a8f9cc6c35ec5e7e243" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 33371 } }
I20260926 15:11:58.770382 25812 ts_tablet_manager.cc:1428] T e4c7e54e5d2c449dbe4a638a69fc6f30 P ed30be0b68124b0ead8c15e6ffe0aef7: Time spent starting tablet: real 0.042s user 0.015s sys 0.001s
I20260926 15:11:58.777241 25815 raft_consensus.cc:348] T e4c7e54e5d2c449dbe4a638a69fc6f30 P 5933f4a56ce04a8f9cc6c35ec5e7e243 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed30be0b68124b0ead8c15e6ffe0aef7" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 34699 } } peers { permanent_uuid: "a26ca453e2c941d2b4d5e9972af91fde" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 33971 } } peers { permanent_uuid: "5933f4a56ce04a8f9cc6c35ec5e7e243" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 33371 } }
I20260926 15:11:58.777840 25815 raft_consensus.cc:374] T e4c7e54e5d2c449dbe4a638a69fc6f30 P 5933f4a56ce04a8f9cc6c35ec5e7e243 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:11:58.777966 25815 raft_consensus.cc:729] T e4c7e54e5d2c449dbe4a638a69fc6f30 P 5933f4a56ce04a8f9cc6c35ec5e7e243 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5933f4a56ce04a8f9cc6c35ec5e7e243, State: Initialized, Role: FOLLOWER
I20260926 15:11:58.778890 25815 consensus_queue.cc:260] T e4c7e54e5d2c449dbe4a638a69fc6f30 P 5933f4a56ce04a8f9cc6c35ec5e7e243 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed30be0b68124b0ead8c15e6ffe0aef7" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 34699 } } peers { permanent_uuid: "a26ca453e2c941d2b4d5e9972af91fde" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 33971 } } peers { permanent_uuid: "5933f4a56ce04a8f9cc6c35ec5e7e243" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 33371 } }
I20260926 15:11:58.789686 25815 ts_tablet_manager.cc:1428] T e4c7e54e5d2c449dbe4a638a69fc6f30 P 5933f4a56ce04a8f9cc6c35ec5e7e243: Time spent starting tablet: real 0.033s user 0.017s sys 0.003s
I20260926 15:11:58.789740 25512 heartbeater.cc:503] Master 127.12.34.254:45857 was elected leader, sending a full tablet report...
I20260926 15:11:58.790045 25790 heartbeater.cc:503] Master 127.12.34.254:45857 was elected leader, sending a full tablet report...
I20260926 15:11:58.790331 25814 ts_tablet_manager.cc:1428] T e4c7e54e5d2c449dbe4a638a69fc6f30 P a26ca453e2c941d2b4d5e9972af91fde: Time spent starting tablet: real 0.035s user 0.020s sys 0.014s
W20260926 15:11:58.859297 25657 tablet.cc:2369] T e4c7e54e5d2c449dbe4a638a69fc6f30 P ed30be0b68124b0ead8c15e6ffe0aef7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 15:11:58.930689 25513 tablet.cc:2369] T e4c7e54e5d2c449dbe4a638a69fc6f30 P a26ca453e2c941d2b4d5e9972af91fde: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 15:11:59.003398 25791 tablet.cc:2369] T e4c7e54e5d2c449dbe4a638a69fc6f30 P 5933f4a56ce04a8f9cc6c35ec5e7e243: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 15:11:59.022063 25818 raft_consensus.cc:482] T e4c7e54e5d2c449dbe4a638a69fc6f30 P ed30be0b68124b0ead8c15e6ffe0aef7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 15:11:59.022388 25818 raft_consensus.cc:504] T e4c7e54e5d2c449dbe4a638a69fc6f30 P ed30be0b68124b0ead8c15e6ffe0aef7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed30be0b68124b0ead8c15e6ffe0aef7" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 34699 } } peers { permanent_uuid: "a26ca453e2c941d2b4d5e9972af91fde" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 33971 } } peers { permanent_uuid: "5933f4a56ce04a8f9cc6c35ec5e7e243" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 33371 } }
I20260926 15:11:59.024158 25818 leader_election.cc:288] T e4c7e54e5d2c449dbe4a638a69fc6f30 P ed30be0b68124b0ead8c15e6ffe0aef7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a26ca453e2c941d2b4d5e9972af91fde (127.12.34.193:33971), 5933f4a56ce04a8f9cc6c35ec5e7e243 (127.12.34.195:33371)
I20260926 15:11:59.032752 25811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:11:59.025289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.194:39937 (local address 127.12.34.195:33371)
0926 15:11:59.025521 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0926 15:11:59.025530 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 15:11:59.026796 (+ 1266us) server_negotiation.cc:407] Connection header received
0926 15:11:59.027023 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:11:59.027028 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:11:59.027105 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:11:59.027193 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:11:59.027855 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:11:59.028860 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:11:59.030726 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:11:59.031799 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:11:59.031970 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:11:59.032049 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 15:11:59.032357 (+ 308us) server_negotiation.cc:299] Negotiation successful
0926 15:11:59.032483 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 15:11:59.033025 25828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:11:59.024435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.195:33371 (local address 127.12.34.194:39937)
0926 15:11:59.026716 (+ 2281us) negotiation.cc:107] Waiting for socket to connect
0926 15:11:59.026734 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 15:11:59.026894 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:11:59.027217 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:11:59.027222 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:11:59.027250 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:11:59.027693 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:11:59.027704 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:11:59.028983 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:11:59.028993 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:11:59.030581 (+ 1588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:11:59.030592 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:11:59.032005 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:11:59.032011 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:11:59.032175 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:11:59.032210 (+ 35us) client_negotiation.cc:769] Sending connection context
0926 15:11:59.032306 (+ 96us) client_negotiation.cc:240] Negotiation successful
0926 15:11:59.032439 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2175,"mutex_wait_us":46,"thread_start_us":1612,"threads_started":1}
I20260926 15:11:59.033646 25810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:11:59.024712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.194:41387 (local address 127.12.34.193:33971)
0926 15:11:59.024958 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0926 15:11:59.024967 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 15:11:59.026541 (+ 1574us) server_negotiation.cc:407] Connection header received
0926 15:11:59.026845 (+ 304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:11:59.026860 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:11:59.026940 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:11:59.027032 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:11:59.027600 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:11:59.028659 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:11:59.031094 (+ 2435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:11:59.032236 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:11:59.032416 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:11:59.032482 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 15:11:59.033262 (+ 780us) server_negotiation.cc:299] Negotiation successful
0926 15:11:59.033417 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 15:11:59.034226 25829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:11:59.024360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.193:33971 (local address 127.12.34.194:41387)
0926 15:11:59.026421 (+ 2061us) negotiation.cc:107] Waiting for socket to connect
0926 15:11:59.026444 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 15:11:59.026662 (+ 218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:11:59.027041 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:11:59.027046 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:11:59.027065 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:11:59.027459 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:11:59.027470 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:11:59.028804 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:11:59.028809 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:11:59.030933 (+ 2124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:11:59.030947 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:11:59.032644 (+ 1697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:11:59.032650 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:11:59.032962 (+ 312us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:11:59.032994 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 15:11:59.033893 (+ 899us) client_negotiation.cc:240] Negotiation successful
0926 15:11:59.033984 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1886,"mutex_wait_us":6,"thread_start_us":2057,"threads_started":1}
I20260926 15:11:59.034169 25745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4c7e54e5d2c449dbe4a638a69fc6f30" candidate_uuid: "ed30be0b68124b0ead8c15e6ffe0aef7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5933f4a56ce04a8f9cc6c35ec5e7e243" is_pre_election: true
I20260926 15:11:59.034718 25745 raft_consensus.cc:2454] T e4c7e54e5d2c449dbe4a638a69fc6f30 P 5933f4a56ce04a8f9cc6c35ec5e7e243 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed30be0b68124b0ead8c15e6ffe0aef7 in term 0.
I20260926 15:11:59.035125 25467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4c7e54e5d2c449dbe4a638a69fc6f30" candidate_uuid: "ed30be0b68124b0ead8c15e6ffe0aef7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a26ca453e2c941d2b4d5e9972af91fde" is_pre_election: true
I20260926 15:11:59.035394 25544 leader_election.cc:302] T e4c7e54e5d2c449dbe4a638a69fc6f30 P ed30be0b68124b0ead8c15e6ffe0aef7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5933f4a56ce04a8f9cc6c35ec5e7e243, ed30be0b68124b0ead8c15e6ffe0aef7; no voters:
I20260926 15:11:59.035622 25467 raft_consensus.cc:2454] T e4c7e54e5d2c449dbe4a638a69fc6f30 P a26ca453e2c941d2b4d5e9972af91fde [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed30be0b68124b0ead8c15e6ffe0aef7 in term 0.
I20260926 15:11:59.035914 25818 raft_consensus.cc:2781] T e4c7e54e5d2c449dbe4a638a69fc6f30 P ed30be0b68124b0ead8c15e6ffe0aef7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 15:11:59.036077 25818 raft_consensus.cc:482] T e4c7e54e5d2c449dbe4a638a69fc6f30 P ed30be0b68124b0ead8c15e6ffe0aef7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 15:11:59.036202 25818 raft_consensus.cc:3037] T e4c7e54e5d2c449dbe4a638a69fc6f30 P ed30be0b68124b0ead8c15e6ffe0aef7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:11:59.039580 25818 raft_consensus.cc:504] T e4c7e54e5d2c449dbe4a638a69fc6f30 P ed30be0b68124b0ead8c15e6ffe0aef7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed30be0b68124b0ead8c15e6ffe0aef7" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 34699 } } peers { permanent_uuid: "a26ca453e2c941d2b4d5e9972af91fde" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 33971 } } peers { permanent_uuid: "5933f4a56ce04a8f9cc6c35ec5e7e243" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 33371 } }
I20260926 15:11:59.040259 25818 leader_election.cc:288] T e4c7e54e5d2c449dbe4a638a69fc6f30 P ed30be0b68124b0ead8c15e6ffe0aef7 [CANDIDATE]: Term 1 election: Requested vote from peers a26ca453e2c941d2b4d5e9972af91fde (127.12.34.193:33971), 5933f4a56ce04a8f9cc6c35ec5e7e243 (127.12.34.195:33371)
I20260926 15:11:59.040635 25467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4c7e54e5d2c449dbe4a638a69fc6f30" candidate_uuid: "ed30be0b68124b0ead8c15e6ffe0aef7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a26ca453e2c941d2b4d5e9972af91fde"
I20260926 15:11:59.040678 25745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4c7e54e5d2c449dbe4a638a69fc6f30" candidate_uuid: "ed30be0b68124b0ead8c15e6ffe0aef7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5933f4a56ce04a8f9cc6c35ec5e7e243"
I20260926 15:11:59.040815 25467 raft_consensus.cc:3037] T e4c7e54e5d2c449dbe4a638a69fc6f30 P a26ca453e2c941d2b4d5e9972af91fde [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:11:59.040869 25745 raft_consensus.cc:3037] T e4c7e54e5d2c449dbe4a638a69fc6f30 P 5933f4a56ce04a8f9cc6c35ec5e7e243 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:11:59.043139 25745 raft_consensus.cc:2454] T e4c7e54e5d2c449dbe4a638a69fc6f30 P 5933f4a56ce04a8f9cc6c35ec5e7e243 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed30be0b68124b0ead8c15e6ffe0aef7 in term 1.
I20260926 15:11:59.043503 25544 leader_election.cc:302] T e4c7e54e5d2c449dbe4a638a69fc6f30 P ed30be0b68124b0ead8c15e6ffe0aef7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5933f4a56ce04a8f9cc6c35ec5e7e243, ed30be0b68124b0ead8c15e6ffe0aef7; no voters:
I20260926 15:11:59.043762 25818 raft_consensus.cc:2781] T e4c7e54e5d2c449dbe4a638a69fc6f30 P ed30be0b68124b0ead8c15e6ffe0aef7 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:11:59.044076 25467 raft_consensus.cc:2454] T e4c7e54e5d2c449dbe4a638a69fc6f30 P a26ca453e2c941d2b4d5e9972af91fde [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed30be0b68124b0ead8c15e6ffe0aef7 in term 1.
I20260926 15:11:59.044207 25818 raft_consensus.cc:686] T e4c7e54e5d2c449dbe4a638a69fc6f30 P ed30be0b68124b0ead8c15e6ffe0aef7 [term 1 LEADER]: Becoming Leader. State: Replica: ed30be0b68124b0ead8c15e6ffe0aef7, State: Running, Role: LEADER
I20260926 15:11:59.044693 25818 consensus_queue.cc:237] T e4c7e54e5d2c449dbe4a638a69fc6f30 P ed30be0b68124b0ead8c15e6ffe0aef7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed30be0b68124b0ead8c15e6ffe0aef7" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 34699 } } peers { permanent_uuid: "a26ca453e2c941d2b4d5e9972af91fde" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 33971 } } peers { permanent_uuid: "5933f4a56ce04a8f9cc6c35ec5e7e243" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 33371 } }
I20260926 15:11:59.049664 25315 catalog_manager.cc:5495] T e4c7e54e5d2c449dbe4a638a69fc6f30 P ed30be0b68124b0ead8c15e6ffe0aef7 reported cstate change: term changed from 0 to 1, leader changed from <none> to ed30be0b68124b0ead8c15e6ffe0aef7 (127.12.34.194). New cstate: current_term: 1 leader_uuid: "ed30be0b68124b0ead8c15e6ffe0aef7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed30be0b68124b0ead8c15e6ffe0aef7" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 34699 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a26ca453e2c941d2b4d5e9972af91fde" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 33971 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5933f4a56ce04a8f9cc6c35ec5e7e243" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 33371 } health_report { overall_health: UNKNOWN } } }
I20260926 15:11:59.125245 25807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:11:59.118276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34948 (local address 127.12.34.194:34699)
0926 15:11:59.118579 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0926 15:11:59.118590 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 15:11:59.119036 (+ 446us) server_negotiation.cc:407] Connection header received
0926 15:11:59.119240 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:11:59.119244 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:11:59.119397 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:11:59.119530 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 15:11:59.120037 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:11:59.120868 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:11:59.122859 (+ 1991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:11:59.123275 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:11:59.123382 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:11:59.123970 (+ 588us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 15:11:59.124520 (+ 550us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 15:11:59.124648 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0926 15:11:59.124768 (+ 120us) server_negotiation.cc:299] Negotiation successful
0926 15:11:59.124839 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 15:11:59.133778 25467 raft_consensus.cc:1261] T e4c7e54e5d2c449dbe4a638a69fc6f30 P a26ca453e2c941d2b4d5e9972af91fde [term 1 FOLLOWER]: Refusing update from remote peer ed30be0b68124b0ead8c15e6ffe0aef7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:11:59.133778 25745 raft_consensus.cc:1261] T e4c7e54e5d2c449dbe4a638a69fc6f30 P 5933f4a56ce04a8f9cc6c35ec5e7e243 [term 1 FOLLOWER]: Refusing update from remote peer ed30be0b68124b0ead8c15e6ffe0aef7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:11:59.134857 25830 consensus_queue.cc:1035] T e4c7e54e5d2c449dbe4a638a69fc6f30 P ed30be0b68124b0ead8c15e6ffe0aef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a26ca453e2c941d2b4d5e9972af91fde" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 33971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:11:59.135215 25818 consensus_queue.cc:1035] T e4c7e54e5d2c449dbe4a638a69fc6f30 P ed30be0b68124b0ead8c15e6ffe0aef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5933f4a56ce04a8f9cc6c35ec5e7e243" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 33371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:11:59.151675 25835 mvcc.cc:205] Tried to move back new op lower bound from 7333623886361059328 to 7333623886011146240. Current Snapshot: MvccSnapshot[applied={T|T < 7333623886361059328}]
I20260926 15:11:59.157521 25834 mvcc.cc:205] Tried to move back new op lower bound from 7333623886361059328 to 7333623886011146240. Current Snapshot: MvccSnapshot[applied={T|T < 7333623886361059328}]
I20260926 15:11:59.172685 25315 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43516:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 15:11:59.172984 25315 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43516:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 15:11:59.177101 25315 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 503ca1bbe012419898d9ac8c1fa73872: Sending DeleteTablet for 3 replicas of tablet e4c7e54e5d2c449dbe4a638a69fc6f30
I20260926 15:11:59.178102 25591 tablet_service.cc:1514] Processing DeleteTablet for tablet e4c7e54e5d2c449dbe4a638a69fc6f30 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:11:59 UTC) from {username='kudu', principal='kudu/127.12.34.254@KRBTEST.COM'} at 127.0.0.1:34934
I20260926 15:11:59.179258 25447 tablet_service.cc:1514] Processing DeleteTablet for tablet e4c7e54e5d2c449dbe4a638a69fc6f30 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:11:59 UTC) from {username='kudu', principal='kudu/127.12.34.254@KRBTEST.COM'} at 127.0.0.1:37466
I20260926 15:11:59.186846 25725 tablet_service.cc:1514] Processing DeleteTablet for tablet e4c7e54e5d2c449dbe4a638a69fc6f30 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:11:59 UTC) from {username='kudu', principal='kudu/127.12.34.254@KRBTEST.COM'} at 127.0.0.1:40192
I20260926 15:11:59.189990 12427 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 25352
I20260926 15:11:59.213954 25847 tablet_replica.cc:330] T e4c7e54e5d2c449dbe4a638a69fc6f30 P 5933f4a56ce04a8f9cc6c35ec5e7e243: stopping tablet replica
I20260926 15:11:59.214455 25847 raft_consensus.cc:2229] T e4c7e54e5d2c449dbe4a638a69fc6f30 P 5933f4a56ce04a8f9cc6c35ec5e7e243 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 15:11:59.214988 25847 raft_consensus.cc:2258] T e4c7e54e5d2c449dbe4a638a69fc6f30 P 5933f4a56ce04a8f9cc6c35ec5e7e243 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:11:59.217319 25847 ts_tablet_manager.cc:1905] T e4c7e54e5d2c449dbe4a638a69fc6f30 P 5933f4a56ce04a8f9cc6c35ec5e7e243: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:11:59.219614 25846 tablet_replica.cc:330] T e4c7e54e5d2c449dbe4a638a69fc6f30 P a26ca453e2c941d2b4d5e9972af91fde: stopping tablet replica
I20260926 15:11:59.219991 25846 raft_consensus.cc:2229] T e4c7e54e5d2c449dbe4a638a69fc6f30 P a26ca453e2c941d2b4d5e9972af91fde [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 15:11:59.220444 25846 raft_consensus.cc:2258] T e4c7e54e5d2c449dbe4a638a69fc6f30 P a26ca453e2c941d2b4d5e9972af91fde [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:11:59.224229 25846 ts_tablet_manager.cc:1905] T e4c7e54e5d2c449dbe4a638a69fc6f30 P a26ca453e2c941d2b4d5e9972af91fde: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:11:59.228500 25847 ts_tablet_manager.cc:1918] T e4c7e54e5d2c449dbe4a638a69fc6f30 P 5933f4a56ce04a8f9cc6c35ec5e7e243: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 15:11:59.228750 25847 log.cc:1196] T e4c7e54e5d2c449dbe4a638a69fc6f30 P 5933f4a56ce04a8f9cc6c35ec5e7e243: Deleting WAL directory at /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/minicluster-data/ts-2/wal/wals/e4c7e54e5d2c449dbe4a638a69fc6f30
I20260926 15:11:59.229470 25847 ts_tablet_manager.cc:1939] T e4c7e54e5d2c449dbe4a638a69fc6f30 P 5933f4a56ce04a8f9cc6c35ec5e7e243: Deleting consensus metadata
I20260926 15:11:59.229522 25845 tablet_replica.cc:330] T e4c7e54e5d2c449dbe4a638a69fc6f30 P ed30be0b68124b0ead8c15e6ffe0aef7: stopping tablet replica
I20260926 15:11:59.229918 25845 raft_consensus.cc:2229] T e4c7e54e5d2c449dbe4a638a69fc6f30 P ed30be0b68124b0ead8c15e6ffe0aef7 [term 1 LEADER]: Raft consensus shutting down.
I20260926 15:11:59.230520 25845 raft_consensus.cc:2258] T e4c7e54e5d2c449dbe4a638a69fc6f30 P ed30be0b68124b0ead8c15e6ffe0aef7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:11:59.231762 25299 catalog_manager.cc:4841] TS 5933f4a56ce04a8f9cc6c35ec5e7e243 (127.12.34.195:33371): tablet e4c7e54e5d2c449dbe4a638a69fc6f30 (table test-table [id=4c55bb836ccd49b39cdebae8fa8cc563]) successfully deleted
I20260926 15:11:59.236562 25845 ts_tablet_manager.cc:1905] T e4c7e54e5d2c449dbe4a638a69fc6f30 P ed30be0b68124b0ead8c15e6ffe0aef7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:11:59.243404 25845 ts_tablet_manager.cc:1918] T e4c7e54e5d2c449dbe4a638a69fc6f30 P ed30be0b68124b0ead8c15e6ffe0aef7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 15:11:59.243610 25845 log.cc:1196] T e4c7e54e5d2c449dbe4a638a69fc6f30 P ed30be0b68124b0ead8c15e6ffe0aef7: Deleting WAL directory at /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/minicluster-data/ts-1/wal/wals/e4c7e54e5d2c449dbe4a638a69fc6f30
I20260926 15:11:59.244132 25845 ts_tablet_manager.cc:1939] T e4c7e54e5d2c449dbe4a638a69fc6f30 P ed30be0b68124b0ead8c15e6ffe0aef7: Deleting consensus metadata
I20260926 15:11:59.245253 25810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:11:59.191273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37470 (local address 127.12.34.193:33971)
0926 15:11:59.191446 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0926 15:11:59.191452 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:11:59.212479 (+ 21027us) server_negotiation.cc:407] Connection header received
0926 15:11:59.212662 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:11:59.212668 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:11:59.212761 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:11:59.212888 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:11:59.217775 (+ 4887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:11:59.219135 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:11:59.222480 (+ 3345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:11:59.223177 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:11:59.223304 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:11:59.234310 (+ 11006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:11:59.234327 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:11:59.234346 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:11:59.234408 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:11:59.238283 (+ 3875us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:11:59.242000 (+ 3717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:11:59.242012 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:11:59.242017 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:11:59.242194 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:11:59.242684 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:11:59.242690 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:11:59.242694 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:11:59.243168 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:11:59.243473 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0926 15:11:59.244272 (+ 799us) server_negotiation.cc:299] Negotiation successful
0926 15:11:59.244523 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 15:11:59.246032 25302 catalog_manager.cc:4841] TS ed30be0b68124b0ead8c15e6ffe0aef7 (127.12.34.194:34699): tablet e4c7e54e5d2c449dbe4a638a69fc6f30 (table test-table [id=4c55bb836ccd49b39cdebae8fa8cc563]) successfully deleted
I20260926 15:11:59.247367 25507 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:11:59.359786 25846 ts_tablet_manager.cc:1918] T e4c7e54e5d2c449dbe4a638a69fc6f30 P a26ca453e2c941d2b4d5e9972af91fde: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 15:11:59.360070 25846 log.cc:1196] T e4c7e54e5d2c449dbe4a638a69fc6f30 P a26ca453e2c941d2b4d5e9972af91fde: Deleting WAL directory at /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790435318278681-12427-0/minicluster-data/ts-0/wal/wals/e4c7e54e5d2c449dbe4a638a69fc6f30
I20260926 15:11:59.360392 12427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8uoJGv/build/asan/bin/kudu with pid 25352
W20260926 15:11:59.383018 25301 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.34.193:33971 (error 108)
I20260926 15:11:59.383705 12427 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 25516
I20260926 15:11:59.385957 25851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:11:59.384453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.193:33971 (local address 127.0.0.1:39474)
0926 15:11:59.385692 (+ 1239us) negotiation.cc:107] Waiting for socket to connect
0926 15:11:59.385791 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.193:33971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1125,"thread_start_us":1003,"threads_started":1}
W20260926 15:11:59.386258 25301 catalog_manager.cc:4584] TS a26ca453e2c941d2b4d5e9972af91fde (127.12.34.193:33971): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e4c7e54e5d2c449dbe4a638a69fc6f30: Network error: Client connection negotiation failed: client connection to 127.12.34.193:33971: connect: Connection refused (error 111)
I20260926 15:11:59.399075 25807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:11:59.384467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48668 (local address 127.12.34.194:34699)
0926 15:11:59.384707 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0926 15:11:59.384717 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 15:11:59.384750 (+ 33us) server_negotiation.cc:407] Connection header received
0926 15:11:59.384947 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:11:59.384955 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:11:59.385112 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:11:59.385250 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:11:59.386922 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:11:59.387728 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:11:59.389575 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:11:59.390042 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:11:59.390146 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:11:59.393314 (+ 3168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:11:59.393324 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:11:59.393337 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:11:59.393377 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:11:59.396295 (+ 2918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:11:59.397058 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:11:59.397065 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:11:59.397067 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:11:59.397337 (+ 270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:11:59.397707 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:11:59.397710 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:11:59.397712 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:11:59.398066 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:11:59.398323 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0926 15:11:59.398603 (+ 280us) server_negotiation.cc:299] Negotiation successful
0926 15:11:59.398759 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 15:11:59.399748 25651 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:11:59.440003 25851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:11:59.439518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.193:33971 (local address 127.0.0.1:39486)
0926 15:11:59.439736 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0926 15:11:59.439852 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.193:33971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260926 15:11:59.441128 25851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:11:59.440676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.193:33971 (local address 127.0.0.1:39496)
0926 15:11:59.440842 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0926 15:11:59.440934 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.193:33971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260926 15:11:59.501677 25851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:11:59.501170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.193:33971 (local address 127.0.0.1:39512)
0926 15:11:59.501387 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0926 15:11:59.501493 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.193:33971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260926 15:11:59.502856 25851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:11:59.502421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.193:33971 (local address 127.0.0.1:39518)
0926 15:11:59.502625 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0926 15:11:59.502715 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.193:33971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260926 15:11:59.513816 12427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8uoJGv/build/asan/bin/kudu with pid 25516
I20260926 15:11:59.532748 12427 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 25660
I20260926 15:11:59.546526 25853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:11:59.533423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36950 (local address 127.12.34.195:33371)
0926 15:11:59.533839 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0926 15:11:59.533843 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:11:59.533860 (+ 17us) server_negotiation.cc:407] Connection header received
0926 15:11:59.533914 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:11:59.533917 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:11:59.533987 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:11:59.534079 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:11:59.535538 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:11:59.536276 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:11:59.537720 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:11:59.538098 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:11:59.538192 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:11:59.540804 (+ 2612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:11:59.540814 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:11:59.540828 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:11:59.540871 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:11:59.543966 (+ 3095us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:11:59.544565 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:11:59.544571 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:11:59.544586 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:11:59.544701 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:11:59.544994 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:11:59.544997 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:11:59.544999 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:11:59.545439 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:11:59.545665 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0926 15:11:59.545947 (+ 282us) server_negotiation.cc:299] Negotiation successful
0926 15:11:59.546159 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":204,"threads_started":1}
I20260926 15:11:59.547170 25785 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:11:59.583932 25851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:11:59.583375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.193:33971 (local address 127.0.0.1:39522)
0926 15:11:59.583642 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0926 15:11:59.583751 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.193:33971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260926 15:11:59.585035 25851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:11:59.584598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.193:33971 (local address 127.0.0.1:39526)
0926 15:11:59.584795 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0926 15:11:59.584890 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.193:33971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260926 15:11:59.634867 12427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8uoJGv/build/asan/bin/kudu with pid 25660
I20260926 15:11:59.651407 12427 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 25282
I20260926 15:11:59.652590 25345 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:11:59.765511 25851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:11:59.765033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.193:33971 (local address 127.0.0.1:39534)
0926 15:11:59.765228 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0926 15:11:59.765320 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.193:33971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260926 15:11:59.765512 12427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8uoJGv/build/asan/bin/kudu with pid 25282
I20260926 15:11:59.786540 12427 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260926 15:12:00.121734 12427 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260926 15:12:00.121940 12427 mini_ranger.cc:66] Stopping Ranger...
I20260926 15:12:00.467767 12427 mini_ranger.cc:68] Stopped Ranger
2026-09-26 15:12:00.468 UTC [24334] LOG: received smart shutdown request
2026-09-26 15:12:00.471 UTC [24334] LOG: background worker "logical replication launcher" (PID 24343) exited with exit code 1
2026-09-26 15:12:00.471 UTC [24338] LOG: shutting down
2026-09-26 15:12:00.645 UTC [24334] LOG: database system is shut down
2026-09-26T15:12:00Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (52456 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790435318278681-12427-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 15:12:00 dist-test-slave-865v krb5kdc[25864](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 15:12:00 dist-test-slave-865v krb5kdc[25864](info): set up 2 sockets
Sep 26 15:12:00 dist-test-slave-865v krb5kdc[25864](info): commencing operation
krb5kdc: starting...
W20260926 15:12:01.917536 12427 mini_kdc.cc:121] Time spent starting KDC: real 1.097s user 0.001s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790435318278681-12427-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790435318278681-12427-0/krb5kdc/test-user.keytab.
Sep 26 15:12:02 dist-test-slave-865v krb5kdc[25864](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435522, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T15:12:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T15:12:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.34.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.34.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.34.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.254.keytab.
Entry for principal kudu/127.12.34.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.254.keytab.
I20260926 15:12:02.149596 12427 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8uoJGv/build/asan/bin/kudu
/tmp/dist-test-task8uoJGv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790435318278681-12427-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790435318278681-12427-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790435318278681-12427-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.34.254:42383
--webserver_interface=127.12.34.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.254.keytab
--principal=kudu/127.12.34.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.34.212:35759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.34.254:42383,127.12.34.253:34755,127.12.34.252:34307
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790435318278681-12427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790435318278681-12427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790435318278681-12427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:12:02.479116 25877 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:12:02.479475 25877 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:12:02.479625 25877 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:12:02.486999 25877 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:12:02.487147 25877 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:12:02.487253 25877 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:12:02.487349 25877 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:12:02.487428 25877 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:12:02.497223 25877 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.34.212:35759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790435318278681-12427-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790435318278681-12427-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.34.254:42383,127.12.34.253:34755,127.12.34.252:34307
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.254.keytab
--principal=kudu/127.12.34.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.34.254:42383
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.34.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790435318278681-12427-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:12:02.498615 25877 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:12:02.500005 25877 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:12:02.512069 25883 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:12:02.512190 25882 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:12:02.512117 25885 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:12:02.512463 25877 server_base.cc:1034] running on GCE node
I20260926 15:12:02.513671 25877 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:12:02.515074 25877 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:12:02.516291 25877 hybrid_clock.cc:648] HybridClock initialized: now 1790435522516239 us; error 36 us; skew 500 ppm
I20260926 15:12:02.516698 25877 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:12:02 dist-test-slave-865v krb5kdc[25864](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435522, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:12:02.524019 25877 init.cc:377] Logged in from keytab as kudu/127.12.34.254@KRBTEST.COM (short username kudu)
I20260926 15:12:02.525671 25877 webserver.cc:460] Webserver started at http://127.12.34.254:42669/ using document root <none> and password file <none>
I20260926 15:12:02.526401 25877 fs_manager.cc:359] Metadata directory not provided
I20260926 15:12:02.526552 25877 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:12:02.526856 25877 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:12:02.529851 25877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790435318278681-12427-0/minicluster-data/master-0/data/instance:
uuid: "0cf1f1a1440c4c548a101585ec788628"
format_stamp: "Formatted at 2026-09-26 15:12:02 on dist-test-slave-865v"
server_key: "1ff423cbccabdce68d3ddba7cc07609f"
server_key_iv: "12fa27790653cb3cbb6a1a1535dab6b0"
server_key_version: "encryptionkey@0"
I20260926 15:12:02.530663 25877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790435318278681-12427-0/minicluster-data/master-0/wal/instance:
uuid: "0cf1f1a1440c4c548a101585ec788628"
format_stamp: "Formatted at 2026-09-26 15:12:02 on dist-test-slave-865v"
server_key: "1ff423cbccabdce68d3ddba7cc07609f"
server_key_iv: "12fa27790653cb3cbb6a1a1535dab6b0"
server_key_version: "encryptionkey@0"
I20260926 15:12:02.535634 25877 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260926 15:12:02.539031 25892 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:12:02.540408 25877 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 15:12:02.540612 25877 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790435318278681-12427-0/minicluster-data/master-0/data,/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790435318278681-12427-0/minicluster-data/master-0/wal
uuid: "0cf1f1a1440c4c548a101585ec788628"
format_stamp: "Formatted at 2026-09-26 15:12:02 on dist-test-slave-865v"
server_key: "1ff423cbccabdce68d3ddba7cc07609f"
server_key_iv: "12fa27790653cb3cbb6a1a1535dab6b0"
server_key_version: "encryptionkey@0"
I20260926 15:12:02.540803 25877 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790435318278681-12427-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790435318278681-12427-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790435318278681-12427-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:12:02.576674 25877 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:12:02.580657 25877 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:12:02.580914 25877 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:12:02.600849 25877 rpc_server.cc:307] RPC server started. Bound to: 127.12.34.254:42383
I20260926 15:12:02.601063 25943 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.34.254:42383 every 8 connection(s)
I20260926 15:12:02.602319 25877 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
I20260926 15:12:02.606267 25944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:12:02.612047 12427 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8uoJGv/build/asan/bin/kudu as pid 25877
I20260926 15:12:02.612275 12427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790435318278681-12427-0/minicluster-data/master-0/wal/instance
I20260926 15:12:02.613173 12427 external_mini_cluster.cc:1428] Setting key 35de09e1e681f6cca717f18de62d4ab5
I20260926 15:12:02.613873 25944 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.34.254" port: 42383 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 15:12:02 dist-test-slave-865v krb5kdc[25864](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435522, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.254@KRBTEST.COM for kudu/127.12.34.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.12.34.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.34.253@KRBTEST.COM" created.
I20260926 15:12:02.639029 25947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:02.619028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51520 (local address 127.12.34.254:42383)
0926 15:12:02.622193 (+ 3165us) server_negotiation.cc:206] Beginning negotiation
0926 15:12:02.622200 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:12:02.622252 (+ 52us) server_negotiation.cc:407] Connection header received
0926 15:12:02.622557 (+ 305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:12:02.622562 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:12:02.622788 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:12:02.623016 (+ 228us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:12:02.624311 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:12:02.625689 (+ 1378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:12:02.627657 (+ 1968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:12:02.628453 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:12:02.628798 (+ 345us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:12:02.632583 (+ 3785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:12:02.632591 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:12:02.632602 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:12:02.632628 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:12:02.635338 (+ 2710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:12:02.635976 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:12:02.635988 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:12:02.635991 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:12:02.636123 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:12:02.636478 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:12:02.636485 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:12:02.636489 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:12:02.636951 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:12:02.637197 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0926 15:12:02.637759 (+ 562us) server_negotiation.cc:299] Negotiation successful
0926 15:12:02.638029 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":34,"server-negotiator.queue_time_us":2951,"thread_start_us":2802,"threads_started":1}
I20260926 15:12:02.639070 25946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:02.617965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:42383 (local address 127.0.0.1:51520)
0926 15:12:02.618884 (+ 919us) negotiation.cc:107] Waiting for socket to connect
0926 15:12:02.618944 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0926 15:12:02.622148 (+ 3204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:12:02.623016 (+ 868us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:12:02.623023 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:12:02.623772 (+ 749us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:12:02.624110 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:12:02.624126 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:12:02.625844 (+ 1718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:12:02.625851 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:12:02.627374 (+ 1523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:12:02.627403 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:12:02.628826 (+ 1423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:12:02.628839 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:12:02.629141 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:12:02.629797 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:12:02.629818 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:12:02.632350 (+ 2532us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:12:02.635462 (+ 3112us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:12:02.635473 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:12:02.635485 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:12:02.635854 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:12:02.636226 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:12:02.636233 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:12:02.636239 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:12:02.636381 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:12:02.637136 (+ 755us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:12:02.637145 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:12:02.637346 (+ 201us) client_negotiation.cc:769] Sending connection context
0926 15:12:02.637632 (+ 286us) client_negotiation.cc:240] Negotiation successful
0926 15:12:02.638189 (+ 557us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"mutex_wait_us":61,"thread_start_us":333,"threads_started":1}
I20260926 15:12:02.642190 25944 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.34.253" port: 34755 } has no permanent_uuid. Determining permanent_uuid...
I20260926 15:12:02.643187 25946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:02.642820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.253:34755 (local address 127.0.0.1:41604)
0926 15:12:02.642988 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0926 15:12:02.643086 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.253:34755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
W20260926 15:12:02.643520 25893 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.34.253:34755: connect: Connection refused (error 111)
I20260926 15:12:02.645044 25946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:02.644682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.253:34755 (local address 127.0.0.1:41616)
0926 15:12:02.644813 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0926 15:12:02.644895 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.253:34755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260926 15:12:02.645288 25944 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.34.253:34755: Network error: Client connection negotiation failed: client connection to 127.12.34.253:34755: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.34.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.253.keytab.
Entry for principal kudu/127.12.34.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.253.keytab.
I20260926 15:12:02.662704 12427 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8uoJGv/build/asan/bin/kudu
/tmp/dist-test-task8uoJGv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790435318278681-12427-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790435318278681-12427-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790435318278681-12427-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790435318278681-12427-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.34.253:34755
--webserver_interface=127.12.34.253
--webserver_port=0
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.253.keytab
--principal=kudu/127.12.34.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.34.212:35759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.34.254:42383,127.12.34.253:34755,127.12.34.252:34307
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790435318278681-12427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790435318278681-12427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790435318278681-12427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 15:12:02.694638 25944 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.34.253" port: 34755 } attempt: 1
I20260926 15:12:02.695940 25946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:02.695366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.253:34755 (local address 127.0.0.1:41624)
0926 15:12:02.695623 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0926 15:12:02.695772 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.253:34755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260926 15:12:02.697122 25946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:02.696682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.253:34755 (local address 127.0.0.1:41628)
0926 15:12:02.696835 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0926 15:12:02.696940 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.253:34755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260926 15:12:02.697320 25944 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.34.253:34755: Network error: Client connection negotiation failed: client connection to 127.12.34.253:34755: connect: Connection refused (error 111)
I20260926 15:12:02.765579 25944 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.34.253" port: 34755 } attempt: 2
I20260926 15:12:02.766917 25946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:02.766332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.253:34755 (local address 127.0.0.1:41634)
0926 15:12:02.766605 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0926 15:12:02.766716 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.253:34755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260926 15:12:02.768263 25946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:02.767695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.253:34755 (local address 127.0.0.1:41638)
0926 15:12:02.767950 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0926 15:12:02.768058 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.253:34755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260926 15:12:02.768527 25944 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.34.253:34755: Network error: Client connection negotiation failed: client connection to 127.12.34.253:34755: connect: Connection refused (error 111)
I20260926 15:12:02.859829 25944 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.34.253" port: 34755 } attempt: 3
I20260926 15:12:02.861179 25946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:02.860545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.253:34755 (local address 127.0.0.1:41644)
0926 15:12:02.860806 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0926 15:12:02.860968 (+ 162us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.253:34755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260926 15:12:02.862346 25946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:02.861903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.253:34755 (local address 127.0.0.1:41660)
0926 15:12:02.862058 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0926 15:12:02.862145 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.253:34755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260926 15:12:02.862596 25944 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.34.253:34755: Network error: Client connection negotiation failed: client connection to 127.12.34.253:34755: connect: Connection refused (error 111)
W20260926 15:12:02.985215 25951 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:12:02.985479 25951 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:12:02.985567 25951 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:12:02.992733 25951 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:12:02.992838 25951 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:12:02.992887 25951 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:12:02.992954 25951 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:12:02.993001 25951 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:12:03.002779 25951 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.34.212:35759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790435318278681-12427-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790435318278681-12427-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.12.34.254:42383,127.12.34.253:34755,127.12.34.252:34307
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.253.keytab
--principal=kudu/127.12.34.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.34.253:34755
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790435318278681-12427-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.12.34.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790435318278681-12427-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:12:03.004068 25951 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:12:03.005286 25951 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 15:12:03.005875 25944 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.34.253" port: 34755 } attempt: 4
I20260926 15:12:03.007144 25946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:03.006500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.253:34755 (local address 127.0.0.1:41674)
0926 15:12:03.006868 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0926 15:12:03.006974 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.253:34755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154}
I20260926 15:12:03.008548 25946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:03.008047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.253:34755 (local address 127.0.0.1:41688)
0926 15:12:03.008245 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0926 15:12:03.008368 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.253:34755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
W20260926 15:12:03.008765 25944 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.34.253:34755: Network error: Client connection negotiation failed: client connection to 127.12.34.253:34755: connect: Connection refused (error 111)
W20260926 15:12:03.018791 25959 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:12:03.018718 25956 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:12:03.018821 25957 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:12:03.019749 25951 server_base.cc:1034] running on GCE node
I20260926 15:12:03.020560 25951 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:12:03.022040 25951 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:12:03.023264 25951 hybrid_clock.cc:648] HybridClock initialized: now 1790435523023207 us; error 53 us; skew 500 ppm
I20260926 15:12:03.023602 25951 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:12:03 dist-test-slave-865v krb5kdc[25864](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435523, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:12:03.026945 25951 init.cc:377] Logged in from keytab as kudu/127.12.34.253@KRBTEST.COM (short username kudu)
I20260926 15:12:03.028501 25951 webserver.cc:460] Webserver started at http://127.12.34.253:34549/ using document root <none> and password file <none>
I20260926 15:12:03.029243 25951 fs_manager.cc:359] Metadata directory not provided
I20260926 15:12:03.029379 25951 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:12:03.029675 25951 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:12:03.032630 25951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790435318278681-12427-0/minicluster-data/master-1/data/instance:
uuid: "530dfa80c2e14626a702a13801f79aca"
format_stamp: "Formatted at 2026-09-26 15:12:03 on dist-test-slave-865v"
server_key: "11383189711cea568510219e05bebc19"
server_key_iv: "0deea69aaf9038d6c21275b28dc37991"
server_key_version: "encryptionkey@0"
I20260926 15:12:03.033514 25951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790435318278681-12427-0/minicluster-data/master-1/wal/instance:
uuid: "530dfa80c2e14626a702a13801f79aca"
format_stamp: "Formatted at 2026-09-26 15:12:03 on dist-test-slave-865v"
server_key: "11383189711cea568510219e05bebc19"
server_key_iv: "0deea69aaf9038d6c21275b28dc37991"
server_key_version: "encryptionkey@0"
I20260926 15:12:03.038822 25951 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 15:12:03.042384 25966 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:12:03.043756 25951 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 15:12:03.043957 25951 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790435318278681-12427-0/minicluster-data/master-1/data,/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790435318278681-12427-0/minicluster-data/master-1/wal
uuid: "530dfa80c2e14626a702a13801f79aca"
format_stamp: "Formatted at 2026-09-26 15:12:03 on dist-test-slave-865v"
server_key: "11383189711cea568510219e05bebc19"
server_key_iv: "0deea69aaf9038d6c21275b28dc37991"
server_key_version: "encryptionkey@0"
I20260926 15:12:03.044230 25951 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790435318278681-12427-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790435318278681-12427-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790435318278681-12427-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:12:03.060393 25951 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:12:03.064579 25951 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:12:03.064865 25951 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:12:03.083371 25951 rpc_server.cc:307] RPC server started. Bound to: 127.12.34.253:34755
I20260926 15:12:03.083457 26017 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.34.253:34755 every 8 connection(s)
I20260926 15:12:03.084687 25951 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790435318278681-12427-0/minicluster-data/master-1/data/info.pb
I20260926 15:12:03.088498 26018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:12:03.093827 12427 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8uoJGv/build/asan/bin/kudu as pid 25951
I20260926 15:12:03.094044 12427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790435318278681-12427-0/minicluster-data/master-1/wal/instance
I20260926 15:12:03.094476 12427 external_mini_cluster.cc:1428] Setting key 3b121ba35b36c07caf3a0bb42f949633
I20260926 15:12:03.095507 26018 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.34.254" port: 42383 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 15:12:03 dist-test-slave-865v krb5kdc[25864](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435523, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.253@KRBTEST.COM for kudu/127.12.34.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.12.34.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.34.252@KRBTEST.COM" created.
I20260926 15:12:03.117702 25947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:03.098311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51526 (local address 127.12.34.254:42383)
0926 15:12:03.098563 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0926 15:12:03.098571 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 15:12:03.099669 (+ 1098us) server_negotiation.cc:407] Connection header received
0926 15:12:03.101847 (+ 2178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:12:03.101856 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:12:03.101957 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:12:03.102080 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:12:03.103524 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:12:03.104683 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:12:03.106394 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:12:03.107174 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:12:03.107316 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:12:03.110911 (+ 3595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:12:03.110921 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:12:03.110928 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:12:03.110988 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:12:03.114651 (+ 3663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:12:03.115384 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:12:03.115388 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:12:03.115391 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:12:03.115513 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:12:03.115874 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:12:03.115877 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:12:03.115880 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:12:03.116249 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:12:03.116371 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0926 15:12:03.117138 (+ 767us) server_negotiation.cc:299] Negotiation successful
0926 15:12:03.117357 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260926 15:12:03.118695 26020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:03.098657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:42383 (local address 127.0.0.1:51526)
0926 15:12:03.099501 (+ 844us) negotiation.cc:107] Waiting for socket to connect
0926 15:12:03.099548 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 15:12:03.101587 (+ 2039us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:12:03.102269 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:12:03.102277 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:12:03.102784 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:12:03.103342 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:12:03.103357 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:12:03.104852 (+ 1495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:12:03.104857 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:12:03.106245 (+ 1388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:12:03.106263 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:12:03.107425 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:12:03.107431 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:12:03.107621 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:12:03.108341 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:12:03.108365 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:12:03.110671 (+ 2306us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:12:03.114783 (+ 4112us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:12:03.114791 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:12:03.114802 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:12:03.115258 (+ 456us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:12:03.115611 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:12:03.115615 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:12:03.115617 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:12:03.115779 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:12:03.116361 (+ 582us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:12:03.116366 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:12:03.116749 (+ 383us) client_negotiation.cc:769] Sending connection context
0926 15:12:03.117019 (+ 270us) client_negotiation.cc:240] Negotiation successful
0926 15:12:03.117536 (+ 517us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"thread_start_us":368,"threads_started":1}
I20260926 15:12:03.120451 26018 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.34.253" port: 34755 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 15:12:03 dist-test-slave-865v krb5kdc[25864](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435523, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.253@KRBTEST.COM for kudu/127.12.34.253@KRBTEST.COM
I20260926 15:12:03.138348 26020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:03.121118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.253:34755 (local address 127.0.0.1:41700)
0926 15:12:03.121363 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0926 15:12:03.121383 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 15:12:03.121485 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:12:03.124045 (+ 2560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:12:03.124051 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:12:03.124550 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:12:03.124682 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:12:03.124690 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:12:03.126770 (+ 2080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:12:03.126774 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:12:03.127637 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:12:03.127649 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:12:03.128874 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:12:03.128878 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:12:03.129119 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:12:03.129703 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:12:03.129720 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:12:03.131639 (+ 1919us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:12:03.136028 (+ 4389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:12:03.136032 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:12:03.136035 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:12:03.136386 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:12:03.136793 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:12:03.136796 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:12:03.136798 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:12:03.136934 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:12:03.137679 (+ 745us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:12:03.137683 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:12:03.137795 (+ 112us) client_negotiation.cc:769] Sending connection context
0926 15:12:03.137903 (+ 108us) client_negotiation.cc:240] Negotiation successful
0926 15:12:03.138090 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"mutex_wait_us":1}
I20260926 15:12:03.138661 26022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:03.121329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41700 (local address 127.12.34.253:34755)
0926 15:12:03.123647 (+ 2318us) server_negotiation.cc:206] Beginning negotiation
0926 15:12:03.123652 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:12:03.123676 (+ 24us) server_negotiation.cc:407] Connection header received
0926 15:12:03.123717 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:12:03.123722 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:12:03.123884 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:12:03.123996 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:12:03.124839 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:12:03.126630 (+ 1791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:12:03.127774 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:12:03.128745 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:12:03.128882 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:12:03.131851 (+ 2969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:12:03.131867 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:12:03.131886 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:12:03.131926 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:12:03.135874 (+ 3948us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:12:03.136515 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:12:03.136524 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:12:03.136528 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:12:03.136683 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:12:03.137047 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:12:03.137055 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:12:03.137060 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:12:03.137552 (+ 492us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:12:03.137766 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0926 15:12:03.138052 (+ 286us) server_negotiation.cc:299] Negotiation successful
0926 15:12:03.138241 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":43,"server-negotiator.queue_time_us":2174,"thread_start_us":2051,"threads_started":1}
I20260926 15:12:03.140856 26018 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.34.252" port: 34307 } has no permanent_uuid. Determining permanent_uuid...
I20260926 15:12:03.142189 26020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:03.141663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.252:34307 (local address 127.0.0.1:56672)
0926 15:12:03.141917 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0926 15:12:03.142035 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.252:34307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260926 15:12:03.142551 25970 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.34.252:34307: connect: Connection refused (error 111)
I20260926 15:12:03.144102 26020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:03.143772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.252:34307 (local address 127.0.0.1:56676)
0926 15:12:03.143908 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0926 15:12:03.144006 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.252:34307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260926 15:12:03.144368 26018 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.34.252:34307: Network error: Client connection negotiation failed: client connection to 127.12.34.252:34307: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.34.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.252.keytab.
Entry for principal kudu/127.12.34.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.252.keytab.
I20260926 15:12:03.148310 12427 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8uoJGv/build/asan/bin/kudu
/tmp/dist-test-task8uoJGv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790435318278681-12427-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790435318278681-12427-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790435318278681-12427-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790435318278681-12427-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.34.252:34307
--webserver_interface=127.12.34.252
--webserver_port=0
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.252.keytab
--principal=kudu/127.12.34.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.34.212:35759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.34.254:42383,127.12.34.253:34755,127.12.34.252:34307
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790435318278681-12427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790435318278681-12427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790435318278681-12427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 15:12:03.193802 26018 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.34.252" port: 34307 } attempt: 1
I20260926 15:12:03.195058 26020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:03.194592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.252:34307 (local address 127.0.0.1:56680)
0926 15:12:03.194785 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0926 15:12:03.194902 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.252:34307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260926 15:12:03.196017 26020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:03.195697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.252:34307 (local address 127.0.0.1:56694)
0926 15:12:03.195836 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0926 15:12:03.195911 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.252:34307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260926 15:12:03.196228 26018 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.34.252:34307: Network error: Client connection negotiation failed: client connection to 127.12.34.252:34307: connect: Connection refused (error 111)
I20260926 15:12:03.264429 26018 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.34.252" port: 34307 } attempt: 2
I20260926 15:12:03.265590 26020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:03.265140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.252:34307 (local address 127.0.0.1:56704)
0926 15:12:03.265340 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0926 15:12:03.265436 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.252:34307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260926 15:12:03.266531 26020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:03.266196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.252:34307 (local address 127.0.0.1:56716)
0926 15:12:03.266333 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0926 15:12:03.266409 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.252:34307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260926 15:12:03.266731 26018 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.34.252:34307: Network error: Client connection negotiation failed: client connection to 127.12.34.252:34307: connect: Connection refused (error 111)
I20260926 15:12:03.308027 25944 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.34.253" port: 34755 } attempt: 5
Sep 26 15:12:03 dist-test-slave-865v krb5kdc[25864](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435522, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.254@KRBTEST.COM for kudu/127.12.34.253@KRBTEST.COM
I20260926 15:12:03.321667 25946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:03.308763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.253:34755 (local address 127.0.0.1:41714)
0926 15:12:03.308979 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0926 15:12:03.309001 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 15:12:03.309168 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:12:03.309527 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:12:03.309532 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:12:03.309952 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:12:03.310133 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:12:03.310141 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:12:03.311114 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:12:03.311121 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:12:03.311957 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:12:03.311968 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:12:03.312710 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:12:03.312716 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:12:03.312898 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:12:03.313541 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:12:03.313560 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:12:03.315545 (+ 1985us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:12:03.319043 (+ 3498us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:12:03.319050 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:12:03.319055 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:12:03.319454 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:12:03.319999 (+ 545us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:12:03.320005 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:12:03.320009 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:12:03.320152 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:12:03.320680 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:12:03.320685 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:12:03.320828 (+ 143us) client_negotiation.cc:769] Sending connection context
0926 15:12:03.320992 (+ 164us) client_negotiation.cc:240] Negotiation successful
0926 15:12:03.321303 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260926 15:12:03.321668 26022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:03.308855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41714 (local address 127.12.34.253:34755)
0926 15:12:03.309056 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0926 15:12:03.309061 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:12:03.309111 (+ 50us) server_negotiation.cc:407] Connection header received
0926 15:12:03.309308 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:12:03.309313 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:12:03.309395 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:12:03.309500 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:12:03.310290 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:12:03.310983 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:12:03.312091 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:12:03.312562 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:12:03.312696 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:12:03.315755 (+ 3059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:12:03.315762 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:12:03.315767 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:12:03.315809 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:12:03.318877 (+ 3068us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:12:03.319681 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:12:03.319690 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:12:03.319694 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:12:03.319867 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:12:03.320258 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:12:03.320264 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:12:03.320267 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:12:03.320555 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:12:03.320703 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0926 15:12:03.321131 (+ 428us) server_negotiation.cc:299] Negotiation successful
0926 15:12:03.321344 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 15:12:03.323127 25944 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.34.252" port: 34307 } has no permanent_uuid. Determining permanent_uuid...
I20260926 15:12:03.324566 25946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:03.324030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.252:34307 (local address 127.0.0.1:56722)
0926 15:12:03.324252 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0926 15:12:03.324369 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.252:34307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260926 15:12:03.325721 25946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:03.325305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.252:34307 (local address 127.0.0.1:56736)
0926 15:12:03.325486 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0926 15:12:03.325589 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.252:34307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260926 15:12:03.325939 25944 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.34.252:34307: Network error: Client connection negotiation failed: client connection to 127.12.34.252:34307: connect: Connection refused (error 111)
I20260926 15:12:03.357931 26018 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.34.252" port: 34307 } attempt: 3
I20260926 15:12:03.359047 26020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:03.358592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.252:34307 (local address 127.0.0.1:56750)
0926 15:12:03.358783 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0926 15:12:03.358883 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.252:34307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260926 15:12:03.360265 26020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:03.359753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.252:34307 (local address 127.0.0.1:56764)
0926 15:12:03.360011 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0926 15:12:03.360114 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.252:34307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132}
W20260926 15:12:03.360466 26018 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.34.252:34307: Network error: Client connection negotiation failed: client connection to 127.12.34.252:34307: connect: Connection refused (error 111)
I20260926 15:12:03.377177 25944 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.34.252" port: 34307 } attempt: 1
I20260926 15:12:03.378410 25946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:03.377823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.252:34307 (local address 127.0.0.1:56774)
0926 15:12:03.378049 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0926 15:12:03.378204 (+ 155us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.252:34307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260926 15:12:03.379448 25946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:03.379081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.252:34307 (local address 127.0.0.1:56782)
0926 15:12:03.379223 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0926 15:12:03.379317 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.252:34307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260926 15:12:03.379685 25944 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.34.252:34307: Network error: Client connection negotiation failed: client connection to 127.12.34.252:34307: connect: Connection refused (error 111)
I20260926 15:12:03.447899 25944 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.34.252" port: 34307 } attempt: 2
I20260926 15:12:03.449223 25946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:03.448597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.252:34307 (local address 127.0.0.1:56792)
0926 15:12:03.448858 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0926 15:12:03.448973 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.252:34307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260926 15:12:03.450285 25946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:03.449915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.252:34307 (local address 127.0.0.1:56808)
0926 15:12:03.450067 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0926 15:12:03.450156 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.252:34307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260926 15:12:03.450506 25944 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.34.252:34307: Network error: Client connection negotiation failed: client connection to 127.12.34.252:34307: connect: Connection refused (error 111)
W20260926 15:12:03.472159 26025 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:12:03.472442 26025 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:12:03.472536 26025 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:12:03.479913 26025 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:12:03.480027 26025 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:12:03.480082 26025 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:12:03.480139 26025 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:12:03.480183 26025 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:12:03.489826 26025 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.34.212:35759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790435318278681-12427-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790435318278681-12427-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.12.34.254:42383,127.12.34.253:34755,127.12.34.252:34307
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.252.keytab
--principal=kudu/127.12.34.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.34.252:34307
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790435318278681-12427-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.12.34.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790435318278681-12427-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:12:03.491003 26025 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:12:03.492185 26025 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 15:12:03.503667 26018 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.34.252" port: 34307 } attempt: 4
I20260926 15:12:03.504921 26020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:03.504341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.252:34307 (local address 127.0.0.1:56820)
0926 15:12:03.504582 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0926 15:12:03.504701 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.252:34307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260926 15:12:03.505800 26030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:12:03.505970 26033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:12:03.505766 26031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:12:03.506222 26020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:03.505788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.252:34307 (local address 127.0.0.1:56824)
0926 15:12:03.505994 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0926 15:12:03.506072 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.252:34307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
W20260926 15:12:03.506441 26018 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.34.252:34307: Network error: Client connection negotiation failed: client connection to 127.12.34.252:34307: connect: Connection refused (error 111)
I20260926 15:12:03.506870 26025 server_base.cc:1034] running on GCE node
I20260926 15:12:03.507651 26025 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:12:03.509008 26025 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:12:03.510251 26025 hybrid_clock.cc:648] HybridClock initialized: now 1790435523510206 us; error 38 us; skew 500 ppm
I20260926 15:12:03.510587 26025 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:12:03 dist-test-slave-865v krb5kdc[25864](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435523, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:12:03.513737 26025 init.cc:377] Logged in from keytab as kudu/127.12.34.252@KRBTEST.COM (short username kudu)
I20260926 15:12:03.515282 26025 webserver.cc:460] Webserver started at http://127.12.34.252:33931/ using document root <none> and password file <none>
I20260926 15:12:03.515961 26025 fs_manager.cc:359] Metadata directory not provided
I20260926 15:12:03.516103 26025 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:12:03.516399 26025 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:12:03.519342 26025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790435318278681-12427-0/minicluster-data/master-2/data/instance:
uuid: "947b651652bf4b8188f90b923df5a853"
format_stamp: "Formatted at 2026-09-26 15:12:03 on dist-test-slave-865v"
server_key: "0105ba6351a09cc016aea04b3edc512e"
server_key_iv: "b40163517ca56888760cc02ac357349d"
server_key_version: "encryptionkey@0"
I20260926 15:12:03.520119 26025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790435318278681-12427-0/minicluster-data/master-2/wal/instance:
uuid: "947b651652bf4b8188f90b923df5a853"
format_stamp: "Formatted at 2026-09-26 15:12:03 on dist-test-slave-865v"
server_key: "0105ba6351a09cc016aea04b3edc512e"
server_key_iv: "b40163517ca56888760cc02ac357349d"
server_key_version: "encryptionkey@0"
I20260926 15:12:03.525038 26025 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260926 15:12:03.528544 26040 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:12:03.529953 26025 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 15:12:03.530120 26025 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790435318278681-12427-0/minicluster-data/master-2/data,/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790435318278681-12427-0/minicluster-data/master-2/wal
uuid: "947b651652bf4b8188f90b923df5a853"
format_stamp: "Formatted at 2026-09-26 15:12:03 on dist-test-slave-865v"
server_key: "0105ba6351a09cc016aea04b3edc512e"
server_key_iv: "b40163517ca56888760cc02ac357349d"
server_key_version: "encryptionkey@0"
I20260926 15:12:03.530316 26025 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790435318278681-12427-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790435318278681-12427-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790435318278681-12427-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:12:03.553453 26025 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:12:03.556738 25944 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.34.252" port: 34307 } attempt: 3
I20260926 15:12:03.557514 26025 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:12:03.557812 26025 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:12:03.557871 25946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:03.557340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.252:34307 (local address 127.0.0.1:56828)
0926 15:12:03.557554 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0926 15:12:03.557662 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.252:34307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260926 15:12:03.558923 25946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:03.558464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.252:34307 (local address 127.0.0.1:56838)
0926 15:12:03.558649 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0926 15:12:03.558749 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.252:34307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
W20260926 15:12:03.559173 25944 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.34.252:34307: Network error: Client connection negotiation failed: client connection to 127.12.34.252:34307: connect: Connection refused (error 111)
I20260926 15:12:03.577735 26025 rpc_server.cc:307] RPC server started. Bound to: 127.12.34.252:34307
I20260926 15:12:03.577901 26091 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.34.252:34307 every 8 connection(s)
I20260926 15:12:03.579120 26025 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790435318278681-12427-0/minicluster-data/master-2/data/info.pb
I20260926 15:12:03.579548 12427 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8uoJGv/build/asan/bin/kudu as pid 26025
I20260926 15:12:03.579757 12427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790435318278681-12427-0/minicluster-data/master-2/wal/instance
I20260926 15:12:03.580346 12427 external_mini_cluster.cc:1428] Setting key 2b2f90497b8ab6ea3c848a6114f67b04
I20260926 15:12:03.583532 26092 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:12:03.591853 26092 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.34.254" port: 42383 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 15:12:03 dist-test-slave-865v krb5kdc[25864](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435522, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.34.252@KRBTEST.COM
Sep 26 15:12:03 dist-test-slave-865v krb5kdc[25864](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435523, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.252@KRBTEST.COM for kudu/127.12.34.254@KRBTEST.COM
I20260926 15:12:03.608031 26095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:03.582272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56840 (local address 127.12.34.252:34307)
0926 15:12:03.583046 (+ 774us) server_negotiation.cc:206] Beginning negotiation
0926 15:12:03.583058 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 15:12:03.583096 (+ 38us) server_negotiation.cc:407] Connection header received
0926 15:12:03.584739 (+ 1643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:12:03.584759 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:12:03.585212 (+ 453us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:12:03.585513 (+ 301us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:12:03.587303 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:12:03.588653 (+ 1350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:12:03.590638 (+ 1985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:12:03.591288 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:12:03.591421 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:12:03.596886 (+ 5465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:12:03.596899 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:12:03.596914 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:12:03.596977 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:12:03.601676 (+ 4699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:12:03.602899 (+ 1223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:12:03.602909 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:12:03.602918 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:12:03.603244 (+ 326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:12:03.603761 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:12:03.603765 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:12:03.603769 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:12:03.605049 (+ 1280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:12:03.605522 (+ 473us) server_negotiation.cc:1035] Waiting for connection context
0926 15:12:03.606061 (+ 539us) server_negotiation.cc:299] Negotiation successful
0926 15:12:03.606557 (+ 496us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":577,"thread_start_us":394,"threads_started":1}
I20260926 15:12:03.613045 25947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:03.594960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51532 (local address 127.12.34.254:42383)
0926 15:12:03.595210 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0926 15:12:03.595217 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:12:03.595792 (+ 575us) server_negotiation.cc:407] Connection header received
0926 15:12:03.595989 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:12:03.595992 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:12:03.596056 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:12:03.596131 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:12:03.597659 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:12:03.598659 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:12:03.601107 (+ 2448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:12:03.602028 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:12:03.602181 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:12:03.606192 (+ 4011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:12:03.606202 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:12:03.606214 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:12:03.606261 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:12:03.609695 (+ 3434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:12:03.610625 (+ 930us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:12:03.610630 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:12:03.610634 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:12:03.610800 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:12:03.611265 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:12:03.611271 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:12:03.611275 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:12:03.611670 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:12:03.611874 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0926 15:12:03.612375 (+ 501us) server_negotiation.cc:299] Negotiation successful
0926 15:12:03.612617 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 15:12:03.613438 26096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:03.594919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:42383 (local address 127.0.0.1:51532)
0926 15:12:03.595641 (+ 722us) negotiation.cc:107] Waiting for socket to connect
0926 15:12:03.595690 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 15:12:03.595850 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:12:03.596232 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:12:03.596239 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:12:03.596944 (+ 705us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:12:03.597497 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:12:03.597509 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:12:03.598906 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:12:03.598911 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:12:03.600806 (+ 1895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:12:03.600829 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:12:03.602315 (+ 1486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:12:03.602324 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:12:03.602514 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:12:03.603247 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:12:03.603265 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:12:03.605917 (+ 2652us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:12:03.609874 (+ 3957us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:12:03.609884 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:12:03.609890 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:12:03.610445 (+ 555us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:12:03.610938 (+ 493us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:12:03.610943 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:12:03.610947 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:12:03.611134 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:12:03.611833 (+ 699us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:12:03.611842 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:12:03.612047 (+ 205us) client_negotiation.cc:769] Sending connection context
0926 15:12:03.612236 (+ 189us) client_negotiation.cc:240] Negotiation successful
0926 15:12:03.612583 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":322,"threads_started":1}
I20260926 15:12:03.614895 26092 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.34.253" port: 34755 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 15:12:03 dist-test-slave-865v krb5kdc[25864](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435522, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.34.254@KRBTEST.COM
Sep 26 15:12:03 dist-test-slave-865v krb5kdc[25864](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435523, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.252@KRBTEST.COM for kudu/127.12.34.253@KRBTEST.COM
I20260926 15:12:03.627929 26097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:03.611728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51546 (local address 127.12.34.254:42383)
0926 15:12:03.612373 (+ 645us) server_negotiation.cc:206] Beginning negotiation
0926 15:12:03.612377 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:12:03.612396 (+ 19us) server_negotiation.cc:407] Connection header received
0926 15:12:03.612433 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:12:03.612435 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:12:03.612489 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:12:03.612606 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:12:03.614066 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:12:03.614926 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:12:03.616565 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:12:03.617237 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:12:03.617378 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:12:03.621867 (+ 4489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:12:03.621872 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:12:03.621875 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:12:03.621907 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:12:03.625209 (+ 3302us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:12:03.625988 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:12:03.625996 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:12:03.626000 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:12:03.626127 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:12:03.626528 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:12:03.626532 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:12:03.626534 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:12:03.626806 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:12:03.627081 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0926 15:12:03.627361 (+ 280us) server_negotiation.cc:299] Negotiation successful
0926 15:12:03.627533 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":527,"thread_start_us":361,"threads_started":1}
I20260926 15:12:03.629810 26022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:03.615697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41720 (local address 127.12.34.253:34755)
0926 15:12:03.615851 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0926 15:12:03.615856 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:12:03.615873 (+ 17us) server_negotiation.cc:407] Connection header received
0926 15:12:03.615931 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:12:03.615935 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:12:03.615996 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:12:03.616080 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:12:03.616895 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:12:03.617883 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:12:03.619147 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:12:03.619638 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:12:03.619784 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:12:03.623645 (+ 3861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:12:03.623650 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:12:03.623653 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:12:03.623685 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:12:03.626349 (+ 2664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:12:03.627024 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:12:03.627030 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:12:03.627035 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:12:03.627192 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:12:03.627631 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:12:03.627637 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:12:03.627641 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:12:03.628224 (+ 583us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:12:03.628492 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0926 15:12:03.629223 (+ 731us) server_negotiation.cc:299] Negotiation successful
0926 15:12:03.629403 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 15:12:03.629920 26096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:03.615540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.253:34755 (local address 127.0.0.1:41720)
0926 15:12:03.615700 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0926 15:12:03.615718 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 15:12:03.615812 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:12:03.616126 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:12:03.616133 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:12:03.616583 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:12:03.616739 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:12:03.616746 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:12:03.618030 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:12:03.618038 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:12:03.618993 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:12:03.619005 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:12:03.619796 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:12:03.619803 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:12:03.620012 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:12:03.620623 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:12:03.620649 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:12:03.623450 (+ 2801us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:12:03.626477 (+ 3027us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:12:03.626484 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:12:03.626490 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:12:03.626893 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:12:03.627324 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:12:03.627331 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:12:03.627334 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:12:03.627510 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:12:03.628608 (+ 1098us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:12:03.628613 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:12:03.628839 (+ 226us) client_negotiation.cc:769] Sending connection context
0926 15:12:03.629038 (+ 199us) client_negotiation.cc:240] Negotiation successful
0926 15:12:03.629471 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260926 15:12:03.630949 26092 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.34.252" port: 34307 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 15:12:03 dist-test-slave-865v krb5kdc[25864](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435523, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.252@KRBTEST.COM for kudu/127.12.34.252@KRBTEST.COM
Sep 26 15:12:03 dist-test-slave-865v krb5kdc[25864](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435522, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.34.253@KRBTEST.COM
I20260926 15:12:03.646097 26095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:03.632065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56848 (local address 127.12.34.252:34307)
0926 15:12:03.632264 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0926 15:12:03.632270 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:12:03.632291 (+ 21us) server_negotiation.cc:407] Connection header received
0926 15:12:03.632341 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:12:03.632345 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:12:03.632413 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:12:03.632529 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:12:03.633435 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:12:03.634415 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:12:03.635853 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:12:03.636656 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:12:03.636795 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:12:03.640398 (+ 3603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:12:03.640406 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:12:03.640419 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:12:03.640464 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:12:03.643328 (+ 2864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:12:03.644002 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:12:03.644011 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:12:03.644016 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:12:03.644156 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:12:03.644605 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:12:03.644613 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:12:03.644616 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:12:03.644984 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:12:03.645181 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0926 15:12:03.645618 (+ 437us) server_negotiation.cc:299] Negotiation successful
0926 15:12:03.645809 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 15:12:03.646663 26096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:03.631813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.252:34307 (local address 127.0.0.1:56848)
0926 15:12:03.631971 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0926 15:12:03.631988 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 15:12:03.632080 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:12:03.632542 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:12:03.632545 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:12:03.633058 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:12:03.633273 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:12:03.633285 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:12:03.634563 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:12:03.634567 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:12:03.635708 (+ 1141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:12:03.635723 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:12:03.636810 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:12:03.636815 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:12:03.637073 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:12:03.637654 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:12:03.637679 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:12:03.640220 (+ 2541us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:12:03.643476 (+ 3256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:12:03.643485 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:12:03.643490 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:12:03.643854 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:12:03.644301 (+ 447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:12:03.644309 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:12:03.644314 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:12:03.644454 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:12:03.645156 (+ 702us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:12:03.645161 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:12:03.645338 (+ 177us) client_negotiation.cc:769] Sending connection context
0926 15:12:03.645485 (+ 147us) client_negotiation.cc:240] Negotiation successful
0926 15:12:03.646107 (+ 622us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260926 15:12:03.647921 26022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:03.630516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41732 (local address 127.12.34.253:34755)
0926 15:12:03.630719 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0926 15:12:03.630725 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:12:03.630745 (+ 20us) server_negotiation.cc:407] Connection header received
0926 15:12:03.630883 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:12:03.630887 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:12:03.630941 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:12:03.631024 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:12:03.632773 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:12:03.633788 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:12:03.635246 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:12:03.635913 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:12:03.636053 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:12:03.641391 (+ 5338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:12:03.641399 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:12:03.641406 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:12:03.641461 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:12:03.644186 (+ 2725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:12:03.644985 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:12:03.644995 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:12:03.644998 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:12:03.645196 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:12:03.645751 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:12:03.645760 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:12:03.645764 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:12:03.646420 (+ 656us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:12:03.646736 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0926 15:12:03.647422 (+ 686us) server_negotiation.cc:299] Negotiation successful
0926 15:12:03.647608 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 15:12:03.652539 26092 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 947b651652bf4b8188f90b923df5a853: Bootstrap starting.
I20260926 15:12:03.652843 12427 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 15:12:03.655797 26092 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 947b651652bf4b8188f90b923df5a853: Neither blocks nor log segments found. Creating new log.
I20260926 15:12:03.656803 26092 log.cc:824] T 00000000000000000000000000000000 P 947b651652bf4b8188f90b923df5a853: Log is configured to *not* fsync() on all Append() calls
I20260926 15:12:03.660338 26092 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 947b651652bf4b8188f90b923df5a853: No bootstrap required, opened a new log
I20260926 15:12:03.672248 26092 raft_consensus.cc:348] T 00000000000000000000000000000000 P 947b651652bf4b8188f90b923df5a853 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf1f1a1440c4c548a101585ec788628" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 42383 } } peers { permanent_uuid: "530dfa80c2e14626a702a13801f79aca" member_type: VOTER last_known_addr { host: "127.12.34.253" port: 34755 } } peers { permanent_uuid: "947b651652bf4b8188f90b923df5a853" member_type: VOTER last_known_addr { host: "127.12.34.252" port: 34307 } }
I20260926 15:12:03.672760 26092 raft_consensus.cc:374] T 00000000000000000000000000000000 P 947b651652bf4b8188f90b923df5a853 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:12:03.672885 26092 raft_consensus.cc:729] T 00000000000000000000000000000000 P 947b651652bf4b8188f90b923df5a853 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 947b651652bf4b8188f90b923df5a853, State: Initialized, Role: FOLLOWER
I20260926 15:12:03.673650 26092 consensus_queue.cc:260] T 00000000000000000000000000000000 P 947b651652bf4b8188f90b923df5a853 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf1f1a1440c4c548a101585ec788628" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 42383 } } peers { permanent_uuid: "530dfa80c2e14626a702a13801f79aca" member_type: VOTER last_known_addr { host: "127.12.34.253" port: 34755 } } peers { permanent_uuid: "947b651652bf4b8188f90b923df5a853" member_type: VOTER last_known_addr { host: "127.12.34.252" port: 34307 } }
I20260926 15:12:03.675783 26106 sys_catalog.cc:455] T 00000000000000000000000000000000 P 947b651652bf4b8188f90b923df5a853 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf1f1a1440c4c548a101585ec788628" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 42383 } } peers { permanent_uuid: "530dfa80c2e14626a702a13801f79aca" member_type: VOTER last_known_addr { host: "127.12.34.253" port: 34755 } } peers { permanent_uuid: "947b651652bf4b8188f90b923df5a853" member_type: VOTER last_known_addr { host: "127.12.34.252" port: 34307 } } }
I20260926 15:12:03.678474 26106 sys_catalog.cc:458] T 00000000000000000000000000000000 P 947b651652bf4b8188f90b923df5a853 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 15:12:03.678495 26095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:03.657645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56856 (local address 127.12.34.252:34307)
0926 15:12:03.658088 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0926 15:12:03.658098 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 15:12:03.658656 (+ 558us) server_negotiation.cc:407] Connection header received
0926 15:12:03.658829 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:12:03.658833 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:12:03.658961 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:12:03.659071 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:12:03.661671 (+ 2600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:12:03.663188 (+ 1517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:12:03.665957 (+ 2769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:12:03.666911 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:12:03.667045 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:12:03.671257 (+ 4212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:12:03.671263 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:12:03.671267 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:12:03.671306 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:12:03.674899 (+ 3593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:12:03.675930 (+ 1031us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:12:03.675936 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:12:03.675940 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:12:03.676088 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:12:03.676570 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:12:03.676579 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:12:03.676582 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:12:03.676948 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:12:03.677230 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0926 15:12:03.677789 (+ 559us) server_negotiation.cc:299] Negotiation successful
0926 15:12:03.678042 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241}
I20260926 15:12:03.678692 26097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:03.658274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51556 (local address 127.12.34.254:42383)
0926 15:12:03.658494 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0926 15:12:03.658500 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:12:03.659041 (+ 541us) server_negotiation.cc:407] Connection header received
0926 15:12:03.659273 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:12:03.659279 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:12:03.659360 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:12:03.659438 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:12:03.661814 (+ 2376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:12:03.662883 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:12:03.665226 (+ 2343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:12:03.665837 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:12:03.665951 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:12:03.669914 (+ 3963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:12:03.669920 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:12:03.669925 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:12:03.669967 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:12:03.674103 (+ 4136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:12:03.675906 (+ 1803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:12:03.675911 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:12:03.675914 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:12:03.676054 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:12:03.677061 (+ 1007us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:12:03.677066 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:12:03.677069 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:12:03.677449 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:12:03.677625 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0926 15:12:03.678105 (+ 480us) server_negotiation.cc:299] Negotiation successful
0926 15:12:03.678317 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 15:12:03.677285 26092 sys_catalog.cc:564] T 00000000000000000000000000000000 P 947b651652bf4b8188f90b923df5a853 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:12:03.689807 26022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:03.659595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41746 (local address 127.12.34.253:34755)
0926 15:12:03.659771 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0926 15:12:03.659777 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:12:03.671017 (+ 11240us) server_negotiation.cc:407] Connection header received
0926 15:12:03.671434 (+ 417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:12:03.671440 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:12:03.671526 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:12:03.671596 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:12:03.673397 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:12:03.674539 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:12:03.676605 (+ 2066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:12:03.677275 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:12:03.677366 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:12:03.681991 (+ 4625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:12:03.681998 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:12:03.682002 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:12:03.682049 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:12:03.685832 (+ 3783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:12:03.686730 (+ 898us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:12:03.686736 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:12:03.686740 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:12:03.686924 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:12:03.687425 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:12:03.687430 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:12:03.687434 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:12:03.687818 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:12:03.688056 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0926 15:12:03.689197 (+ 1141us) server_negotiation.cc:299] Negotiation successful
0926 15:12:03.689449 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260926 15:12:03.690287 26117 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 947b651652bf4b8188f90b923df5a853: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 15:12:03.690482 26117 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 15:12:03.686462 26025 master_runner.cc:186] Error getting master registration for 127.12.34.254:42383: OK, instance_id { permanent_uuid: "0cf1f1a1440c4c548a101585ec788628" instance_seqno: 1790435522594449 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 15:12:03.699661 26025 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 15:12:03.736387 25944 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.34.252" port: 34307 } attempt: 4
Sep 26 15:12:03 dist-test-slave-865v krb5kdc[25864](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435522, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.254@KRBTEST.COM for kudu/127.12.34.252@KRBTEST.COM
I20260926 15:12:03.751748 25946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:03.736944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.252:34307 (local address 127.0.0.1:56872)
0926 15:12:03.737171 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0926 15:12:03.737192 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 15:12:03.737297 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:12:03.737568 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:12:03.737573 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:12:03.738116 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:12:03.738324 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:12:03.738335 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:12:03.739381 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:12:03.739386 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:12:03.740829 (+ 1443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:12:03.740843 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:12:03.741678 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:12:03.741685 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:12:03.741875 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:12:03.742505 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:12:03.742529 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:12:03.745154 (+ 2625us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:12:03.749530 (+ 4376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:12:03.749533 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:12:03.749536 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:12:03.749917 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:12:03.750316 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:12:03.750319 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:12:03.750321 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:12:03.750441 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:12:03.751042 (+ 601us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:12:03.751045 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:12:03.751176 (+ 131us) client_negotiation.cc:769] Sending connection context
0926 15:12:03.751284 (+ 108us) client_negotiation.cc:240] Negotiation successful
0926 15:12:03.751478 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20260926 15:12:03.752000 26095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:03.737157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56872 (local address 127.12.34.252:34307)
0926 15:12:03.737315 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0926 15:12:03.737319 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:12:03.737338 (+ 19us) server_negotiation.cc:407] Connection header received
0926 15:12:03.737385 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:12:03.737389 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:12:03.737441 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:12:03.737515 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:12:03.738477 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:12:03.739255 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:12:03.740996 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:12:03.741522 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:12:03.741634 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:12:03.745366 (+ 3732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:12:03.745374 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:12:03.745379 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:12:03.745417 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:12:03.749394 (+ 3977us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:12:03.750024 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:12:03.750031 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:12:03.750036 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:12:03.750205 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:12:03.750537 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:12:03.750542 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:12:03.750546 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:12:03.750904 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:12:03.751061 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0926 15:12:03.751402 (+ 341us) server_negotiation.cc:299] Negotiation successful
0926 15:12:03.751610 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 15:12:03.756526 25944 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0cf1f1a1440c4c548a101585ec788628: Bootstrap starting.
I20260926 15:12:03.760128 25944 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0cf1f1a1440c4c548a101585ec788628: Neither blocks nor log segments found. Creating new log.
I20260926 15:12:03.761171 25944 log.cc:824] T 00000000000000000000000000000000 P 0cf1f1a1440c4c548a101585ec788628: Log is configured to *not* fsync() on all Append() calls
I20260926 15:12:03.763727 25944 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0cf1f1a1440c4c548a101585ec788628: No bootstrap required, opened a new log
I20260926 15:12:03.770521 25944 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0cf1f1a1440c4c548a101585ec788628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf1f1a1440c4c548a101585ec788628" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 42383 } } peers { permanent_uuid: "530dfa80c2e14626a702a13801f79aca" member_type: VOTER last_known_addr { host: "127.12.34.253" port: 34755 } } peers { permanent_uuid: "947b651652bf4b8188f90b923df5a853" member_type: VOTER last_known_addr { host: "127.12.34.252" port: 34307 } }
I20260926 15:12:03.770828 25944 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0cf1f1a1440c4c548a101585ec788628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:12:03.770951 25944 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0cf1f1a1440c4c548a101585ec788628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cf1f1a1440c4c548a101585ec788628, State: Initialized, Role: FOLLOWER
I20260926 15:12:03.771435 25944 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0cf1f1a1440c4c548a101585ec788628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf1f1a1440c4c548a101585ec788628" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 42383 } } peers { permanent_uuid: "530dfa80c2e14626a702a13801f79aca" member_type: VOTER last_known_addr { host: "127.12.34.253" port: 34755 } } peers { permanent_uuid: "947b651652bf4b8188f90b923df5a853" member_type: VOTER last_known_addr { host: "127.12.34.252" port: 34307 } }
I20260926 15:12:03.772311 26120 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0cf1f1a1440c4c548a101585ec788628 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf1f1a1440c4c548a101585ec788628" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 42383 } } peers { permanent_uuid: "530dfa80c2e14626a702a13801f79aca" member_type: VOTER last_known_addr { host: "127.12.34.253" port: 34755 } } peers { permanent_uuid: "947b651652bf4b8188f90b923df5a853" member_type: VOTER last_known_addr { host: "127.12.34.252" port: 34307 } } }
I20260926 15:12:03.772718 26120 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0cf1f1a1440c4c548a101585ec788628 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 15:12:03.773663 25944 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0cf1f1a1440c4c548a101585ec788628 [sys.catalog]: configured and running, proceeding with master startup.
W20260926 15:12:03.783352 26131 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0cf1f1a1440c4c548a101585ec788628: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 15:12:03.783512 26131 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 15:12:03.782491 25877 master_runner.cc:186] Error getting master registration for 127.12.34.253:34755: OK, instance_id { permanent_uuid: "530dfa80c2e14626a702a13801f79aca" instance_seqno: 1790435523078058 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 15:12:03.794983 25877 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 15:12:03.805658 26018 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.34.252" port: 34307 } attempt: 5
W20260926 15:12:03.810328 26046 tablet.cc:2369] T 00000000000000000000000000000000 P 947b651652bf4b8188f90b923df5a853: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 26 15:12:03 dist-test-slave-865v krb5kdc[25864](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435523, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.253@KRBTEST.COM for kudu/127.12.34.252@KRBTEST.COM
I20260926 15:12:03.822570 26020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:03.806253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.252:34307 (local address 127.0.0.1:56876)
0926 15:12:03.806514 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0926 15:12:03.806533 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 15:12:03.806640 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:12:03.807022 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:12:03.807027 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:12:03.807541 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:12:03.807749 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:12:03.807761 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:12:03.809269 (+ 1508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:12:03.809275 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:12:03.810700 (+ 1425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:12:03.810715 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:12:03.811799 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:12:03.811806 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:12:03.812025 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:12:03.812706 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:12:03.812730 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:12:03.815550 (+ 2820us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:12:03.819724 (+ 4174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:12:03.819729 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:12:03.819733 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:12:03.820260 (+ 527us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:12:03.820687 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:12:03.820692 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:12:03.820696 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:12:03.820881 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:12:03.821548 (+ 667us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:12:03.821553 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:12:03.821713 (+ 160us) client_negotiation.cc:769] Sending connection context
0926 15:12:03.821882 (+ 169us) client_negotiation.cc:240] Negotiation successful
0926 15:12:03.822184 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137}
I20260926 15:12:03.822979 26095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:03.806339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56876 (local address 127.12.34.252:34307)
0926 15:12:03.806532 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0926 15:12:03.806538 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:12:03.806587 (+ 49us) server_negotiation.cc:407] Connection header received
0926 15:12:03.806769 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:12:03.806775 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:12:03.806858 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:12:03.806977 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:12:03.807907 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:12:03.809140 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:12:03.810927 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:12:03.811636 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:12:03.811790 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:12:03.815738 (+ 3948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:12:03.815744 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:12:03.815749 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:12:03.815788 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:12:03.819559 (+ 3771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:12:03.820400 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:12:03.820406 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:12:03.820409 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:12:03.820541 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:12:03.821030 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:12:03.821036 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:12:03.821039 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:12:03.821410 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:12:03.821585 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0926 15:12:03.822005 (+ 420us) server_negotiation.cc:299] Negotiation successful
0926 15:12:03.822243 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260926 15:12:03.829823 26018 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 530dfa80c2e14626a702a13801f79aca: Bootstrap starting.
W20260926 15:12:03.833783 25898 tablet.cc:2369] T 00000000000000000000000000000000 P 0cf1f1a1440c4c548a101585ec788628: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 15:12:03.833917 26018 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 530dfa80c2e14626a702a13801f79aca: Neither blocks nor log segments found. Creating new log.
I20260926 15:12:03.835215 26018 log.cc:824] T 00000000000000000000000000000000 P 530dfa80c2e14626a702a13801f79aca: Log is configured to *not* fsync() on all Append() calls
I20260926 15:12:03.838694 26018 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 530dfa80c2e14626a702a13801f79aca: No bootstrap required, opened a new log
I20260926 15:12:03.849562 26018 raft_consensus.cc:348] T 00000000000000000000000000000000 P 530dfa80c2e14626a702a13801f79aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf1f1a1440c4c548a101585ec788628" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 42383 } } peers { permanent_uuid: "530dfa80c2e14626a702a13801f79aca" member_type: VOTER last_known_addr { host: "127.12.34.253" port: 34755 } } peers { permanent_uuid: "947b651652bf4b8188f90b923df5a853" member_type: VOTER last_known_addr { host: "127.12.34.252" port: 34307 } }
I20260926 15:12:03.849951 26018 raft_consensus.cc:374] T 00000000000000000000000000000000 P 530dfa80c2e14626a702a13801f79aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:12:03.850088 26018 raft_consensus.cc:729] T 00000000000000000000000000000000 P 530dfa80c2e14626a702a13801f79aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 530dfa80c2e14626a702a13801f79aca, State: Initialized, Role: FOLLOWER
I20260926 15:12:03.850687 26018 consensus_queue.cc:260] T 00000000000000000000000000000000 P 530dfa80c2e14626a702a13801f79aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf1f1a1440c4c548a101585ec788628" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 42383 } } peers { permanent_uuid: "530dfa80c2e14626a702a13801f79aca" member_type: VOTER last_known_addr { host: "127.12.34.253" port: 34755 } } peers { permanent_uuid: "947b651652bf4b8188f90b923df5a853" member_type: VOTER last_known_addr { host: "127.12.34.252" port: 34307 } }
I20260926 15:12:03.851621 26134 sys_catalog.cc:455] T 00000000000000000000000000000000 P 530dfa80c2e14626a702a13801f79aca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf1f1a1440c4c548a101585ec788628" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 42383 } } peers { permanent_uuid: "530dfa80c2e14626a702a13801f79aca" member_type: VOTER last_known_addr { host: "127.12.34.253" port: 34755 } } peers { permanent_uuid: "947b651652bf4b8188f90b923df5a853" member_type: VOTER last_known_addr { host: "127.12.34.252" port: 34307 } } }
I20260926 15:12:03.852063 26134 sys_catalog.cc:458] T 00000000000000000000000000000000 P 530dfa80c2e14626a702a13801f79aca [sys.catalog]: This master's current role is: FOLLOWER
I20260926 15:12:03.852826 26018 sys_catalog.cc:564] T 00000000000000000000000000000000 P 530dfa80c2e14626a702a13801f79aca [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:12:03.878705 25951 master_runner.cc:263] No leader master found from master 530dfa80c2e14626a702a13801f79aca
I20260926 15:12:03.878862 25951 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260926 15:12:03.887553 26145 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 530dfa80c2e14626a702a13801f79aca: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 15:12:03.887699 26145 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 15:12:03.936568 26120 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0cf1f1a1440c4c548a101585ec788628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 15:12:03.936923 26120 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0cf1f1a1440c4c548a101585ec788628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf1f1a1440c4c548a101585ec788628" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 42383 } } peers { permanent_uuid: "530dfa80c2e14626a702a13801f79aca" member_type: VOTER last_known_addr { host: "127.12.34.253" port: 34755 } } peers { permanent_uuid: "947b651652bf4b8188f90b923df5a853" member_type: VOTER last_known_addr { host: "127.12.34.252" port: 34307 } }
I20260926 15:12:03.938542 26120 leader_election.cc:288] T 00000000000000000000000000000000 P 0cf1f1a1440c4c548a101585ec788628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 530dfa80c2e14626a702a13801f79aca (127.12.34.253:34755), 947b651652bf4b8188f90b923df5a853 (127.12.34.252:34307)
I20260926 15:12:03.939625 25993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0cf1f1a1440c4c548a101585ec788628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "530dfa80c2e14626a702a13801f79aca" is_pre_election: true
I20260926 15:12:03.940227 25993 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 530dfa80c2e14626a702a13801f79aca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0cf1f1a1440c4c548a101585ec788628 in term 0.
I20260926 15:12:03.941380 26067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0cf1f1a1440c4c548a101585ec788628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "947b651652bf4b8188f90b923df5a853" is_pre_election: true
I20260926 15:12:03.941952 26067 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 947b651652bf4b8188f90b923df5a853 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0cf1f1a1440c4c548a101585ec788628 in term 0.
I20260926 15:12:03.942575 25896 leader_election.cc:302] T 00000000000000000000000000000000 P 0cf1f1a1440c4c548a101585ec788628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cf1f1a1440c4c548a101585ec788628, 947b651652bf4b8188f90b923df5a853; no voters:
I20260926 15:12:03.943485 26120 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0cf1f1a1440c4c548a101585ec788628 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 15:12:03.943635 26120 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0cf1f1a1440c4c548a101585ec788628 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 15:12:03.943750 26120 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0cf1f1a1440c4c548a101585ec788628 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:12:03.952467 26120 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0cf1f1a1440c4c548a101585ec788628 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf1f1a1440c4c548a101585ec788628" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 42383 } } peers { permanent_uuid: "530dfa80c2e14626a702a13801f79aca" member_type: VOTER last_known_addr { host: "127.12.34.253" port: 34755 } } peers { permanent_uuid: "947b651652bf4b8188f90b923df5a853" member_type: VOTER last_known_addr { host: "127.12.34.252" port: 34307 } }
I20260926 15:12:03.955489 25993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0cf1f1a1440c4c548a101585ec788628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "530dfa80c2e14626a702a13801f79aca"
I20260926 15:12:03.955708 25993 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 530dfa80c2e14626a702a13801f79aca [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:12:03.959298 25993 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 530dfa80c2e14626a702a13801f79aca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cf1f1a1440c4c548a101585ec788628 in term 1.
I20260926 15:12:03.959729 25893 leader_election.cc:302] T 00000000000000000000000000000000 P 0cf1f1a1440c4c548a101585ec788628 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cf1f1a1440c4c548a101585ec788628, 530dfa80c2e14626a702a13801f79aca; no voters:
I20260926 15:12:03.963469 26067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0cf1f1a1440c4c548a101585ec788628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "947b651652bf4b8188f90b923df5a853"
I20260926 15:12:03.963725 26067 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 947b651652bf4b8188f90b923df5a853 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:12:03.967270 26067 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 947b651652bf4b8188f90b923df5a853 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cf1f1a1440c4c548a101585ec788628 in term 1.
I20260926 15:12:03.967973 26120 leader_election.cc:288] T 00000000000000000000000000000000 P 0cf1f1a1440c4c548a101585ec788628 [CANDIDATE]: Term 1 election: Requested vote from peers 530dfa80c2e14626a702a13801f79aca (127.12.34.253:34755), 947b651652bf4b8188f90b923df5a853 (127.12.34.252:34307)
I20260926 15:12:03.968189 26120 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0cf1f1a1440c4c548a101585ec788628 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:12:03.968335 26120 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0cf1f1a1440c4c548a101585ec788628 [term 1 LEADER]: Becoming Leader. State: Replica: 0cf1f1a1440c4c548a101585ec788628, State: Running, Role: LEADER
I20260926 15:12:03.968888 26120 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0cf1f1a1440c4c548a101585ec788628 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf1f1a1440c4c548a101585ec788628" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 42383 } } peers { permanent_uuid: "530dfa80c2e14626a702a13801f79aca" member_type: VOTER last_known_addr { host: "127.12.34.253" port: 34755 } } peers { permanent_uuid: "947b651652bf4b8188f90b923df5a853" member_type: VOTER last_known_addr { host: "127.12.34.252" port: 34307 } }
I20260926 15:12:03.987166 26147 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0cf1f1a1440c4c548a101585ec788628 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0cf1f1a1440c4c548a101585ec788628. Latest consensus state: current_term: 1 leader_uuid: "0cf1f1a1440c4c548a101585ec788628" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf1f1a1440c4c548a101585ec788628" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 42383 } } peers { permanent_uuid: "530dfa80c2e14626a702a13801f79aca" member_type: VOTER last_known_addr { host: "127.12.34.253" port: 34755 } } peers { permanent_uuid: "947b651652bf4b8188f90b923df5a853" member_type: VOTER last_known_addr { host: "127.12.34.252" port: 34307 } } }
I20260926 15:12:03.987598 26147 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0cf1f1a1440c4c548a101585ec788628 [sys.catalog]: This master's current role is: LEADER
I20260926 15:12:03.994421 26149 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:12:03.996634 26149 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 15:12:04.016019 26067 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 947b651652bf4b8188f90b923df5a853 [term 1 FOLLOWER]: Refusing update from remote peer 0cf1f1a1440c4c548a101585ec788628: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:12:04.017067 26147 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0cf1f1a1440c4c548a101585ec788628 [LEADER]: Connected to new peer: Peer: permanent_uuid: "947b651652bf4b8188f90b923df5a853" member_type: VOTER last_known_addr { host: "127.12.34.252" port: 34307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:12:04.024814 25993 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 530dfa80c2e14626a702a13801f79aca [term 1 FOLLOWER]: Refusing update from remote peer 0cf1f1a1440c4c548a101585ec788628: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:12:04.025736 26147 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0cf1f1a1440c4c548a101585ec788628 [LEADER]: Connected to new peer: Peer: permanent_uuid: "530dfa80c2e14626a702a13801f79aca" member_type: VOTER last_known_addr { host: "127.12.34.253" port: 34755 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:12:04.045454 26106 sys_catalog.cc:455] T 00000000000000000000000000000000 P 947b651652bf4b8188f90b923df5a853 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0cf1f1a1440c4c548a101585ec788628. Latest consensus state: current_term: 1 leader_uuid: "0cf1f1a1440c4c548a101585ec788628" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf1f1a1440c4c548a101585ec788628" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 42383 } } peers { permanent_uuid: "530dfa80c2e14626a702a13801f79aca" member_type: VOTER last_known_addr { host: "127.12.34.253" port: 34755 } } peers { permanent_uuid: "947b651652bf4b8188f90b923df5a853" member_type: VOTER last_known_addr { host: "127.12.34.252" port: 34307 } } }
I20260926 15:12:04.045778 26106 sys_catalog.cc:458] T 00000000000000000000000000000000 P 947b651652bf4b8188f90b923df5a853 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 15:12:04.049146 26150 mvcc.cc:205] Tried to move back new op lower bound from 7333623906359173120 to 7333623906190319616. Current Snapshot: MvccSnapshot[applied={T|T < 7333623906359173120}]
I20260926 15:12:04.051265 26134 sys_catalog.cc:455] T 00000000000000000000000000000000 P 530dfa80c2e14626a702a13801f79aca [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0cf1f1a1440c4c548a101585ec788628. Latest consensus state: current_term: 1 leader_uuid: "0cf1f1a1440c4c548a101585ec788628" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf1f1a1440c4c548a101585ec788628" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 42383 } } peers { permanent_uuid: "530dfa80c2e14626a702a13801f79aca" member_type: VOTER last_known_addr { host: "127.12.34.253" port: 34755 } } peers { permanent_uuid: "947b651652bf4b8188f90b923df5a853" member_type: VOTER last_known_addr { host: "127.12.34.252" port: 34307 } } }
I20260926 15:12:04.051546 26134 sys_catalog.cc:458] T 00000000000000000000000000000000 P 530dfa80c2e14626a702a13801f79aca [sys.catalog]: This master's current role is: FOLLOWER
I20260926 15:12:04.060556 26151 mvcc.cc:205] Tried to move back new op lower bound from 7333623906359173120 to 7333623906190319616. Current Snapshot: MvccSnapshot[applied={T|T < 7333623906359173120}]
I20260926 15:12:04.062067 26134 sys_catalog.cc:455] T 00000000000000000000000000000000 P 530dfa80c2e14626a702a13801f79aca [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0cf1f1a1440c4c548a101585ec788628" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf1f1a1440c4c548a101585ec788628" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 42383 } } peers { permanent_uuid: "530dfa80c2e14626a702a13801f79aca" member_type: VOTER last_known_addr { host: "127.12.34.253" port: 34755 } } peers { permanent_uuid: "947b651652bf4b8188f90b923df5a853" member_type: VOTER last_known_addr { host: "127.12.34.252" port: 34307 } } }
I20260926 15:12:04.062327 26134 sys_catalog.cc:458] T 00000000000000000000000000000000 P 530dfa80c2e14626a702a13801f79aca [sys.catalog]: This master's current role is: FOLLOWER
I20260926 15:12:04.074162 26120 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0cf1f1a1440c4c548a101585ec788628 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0cf1f1a1440c4c548a101585ec788628" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf1f1a1440c4c548a101585ec788628" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 42383 } } peers { permanent_uuid: "530dfa80c2e14626a702a13801f79aca" member_type: VOTER last_known_addr { host: "127.12.34.253" port: 34755 } } peers { permanent_uuid: "947b651652bf4b8188f90b923df5a853" member_type: VOTER last_known_addr { host: "127.12.34.252" port: 34307 } } }
I20260926 15:12:04.074525 26120 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0cf1f1a1440c4c548a101585ec788628 [sys.catalog]: This master's current role is: LEADER
I20260926 15:12:04.074848 26120 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0cf1f1a1440c4c548a101585ec788628 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0cf1f1a1440c4c548a101585ec788628" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf1f1a1440c4c548a101585ec788628" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 42383 } } peers { permanent_uuid: "530dfa80c2e14626a702a13801f79aca" member_type: VOTER last_known_addr { host: "127.12.34.253" port: 34755 } } peers { permanent_uuid: "947b651652bf4b8188f90b923df5a853" member_type: VOTER last_known_addr { host: "127.12.34.252" port: 34307 } } }
I20260926 15:12:04.075120 26120 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0cf1f1a1440c4c548a101585ec788628 [sys.catalog]: This master's current role is: LEADER
I20260926 15:12:04.077584 26149 catalog_manager.cc:1347] Generated new cluster ID: 41fbeeb3402b4e52bc6fdb6404dcaf00
I20260926 15:12:04.077679 26149 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 15:12:04.080413 26106 sys_catalog.cc:455] T 00000000000000000000000000000000 P 947b651652bf4b8188f90b923df5a853 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0cf1f1a1440c4c548a101585ec788628" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf1f1a1440c4c548a101585ec788628" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 42383 } } peers { permanent_uuid: "530dfa80c2e14626a702a13801f79aca" member_type: VOTER last_known_addr { host: "127.12.34.253" port: 34755 } } peers { permanent_uuid: "947b651652bf4b8188f90b923df5a853" member_type: VOTER last_known_addr { host: "127.12.34.252" port: 34307 } } }
I20260926 15:12:04.080653 26106 sys_catalog.cc:458] T 00000000000000000000000000000000 P 947b651652bf4b8188f90b923df5a853 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 15:12:04.129280 26149 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 15:12:04.131237 26149 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:12:04.155687 26149 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0cf1f1a1440c4c548a101585ec788628: Generated new TSK 0
I20260926 15:12:04.156821 26149 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 15:12:04.692555 26117 catalog_manager.cc:1259] Loaded cluster ID: 41fbeeb3402b4e52bc6fdb6404dcaf00
I20260926 15:12:04.692718 26117 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 947b651652bf4b8188f90b923df5a853: loading cluster ID for follower catalog manager: success
I20260926 15:12:04.695148 26117 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 947b651652bf4b8188f90b923df5a853: acquiring CA information for follower catalog manager: success
I20260926 15:12:04.696854 26117 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 947b651652bf4b8188f90b923df5a853: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 15:12:04.889780 26145 catalog_manager.cc:1259] Loaded cluster ID: 41fbeeb3402b4e52bc6fdb6404dcaf00
I20260926 15:12:04.889930 26145 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 530dfa80c2e14626a702a13801f79aca: loading cluster ID for follower catalog manager: success
I20260926 15:12:04.892314 26145 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 530dfa80c2e14626a702a13801f79aca: acquiring CA information for follower catalog manager: success
I20260926 15:12:04.894109 26145 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 530dfa80c2e14626a702a13801f79aca: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 15:12:05.269575 12427 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 25877
I20260926 15:12:05.270905 25939 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:12:05.380429 12427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8uoJGv/build/asan/bin/kudu with pid 25877
W20260926 15:12:05.401221 26042 connection.cc:535] server connection from 127.0.0.1:56872 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 15:12:05.401666 12427 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 25951
I20260926 15:12:05.402817 26013 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:12:05.508746 12427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8uoJGv/build/asan/bin/kudu with pid 25951
I20260926 15:12:05.529254 12427 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 26025
I20260926 15:12:05.530328 26087 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:12:05.637753 12427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8uoJGv/build/asan/bin/kudu with pid 26025
2026-09-26T15:12:05Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4860 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790435318278681-12427-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 15:12:05 dist-test-slave-865v krb5kdc[26166](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 15:12:05 dist-test-slave-865v krb5kdc[26166](info): set up 2 sockets
Sep 26 15:12:05 dist-test-slave-865v krb5kdc[26166](info): commencing operation
krb5kdc: starting...
W20260926 15:12:06.766549 12427 mini_kdc.cc:121] Time spent starting KDC: real 1.085s user 0.001s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790435318278681-12427-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790435318278681-12427-0/krb5kdc/test-user.keytab.
Sep 26 15:12:06 dist-test-slave-865v krb5kdc[26166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435526, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T15:12:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T15:12:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.34.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.34.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.34.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.254.keytab.
Entry for principal kudu/127.12.34.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.254.keytab.
I20260926 15:12:07.004163 12427 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8uoJGv/build/asan/bin/kudu
/tmp/dist-test-task8uoJGv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790435318278681-12427-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790435318278681-12427-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790435318278681-12427-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.34.254:46751
--webserver_interface=127.12.34.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.254.keytab
--principal=kudu/127.12.34.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.34.212:40031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.34.254:46751
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790435318278681-12427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790435318278681-12427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790435318278681-12427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:12:07.329643 26179 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:12:07.329959 26179 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:12:07.330113 26179 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:12:07.337253 26179 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:12:07.337378 26179 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:12:07.337481 26179 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:12:07.337585 26179 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:12:07.337667 26179 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:12:07.347666 26179 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.34.212:40031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790435318278681-12427-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790435318278681-12427-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.34.254:46751
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.254.keytab
--principal=kudu/127.12.34.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.34.254:46751
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.34.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790435318278681-12427-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:12:07.348892 26179 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:12:07.350111 26179 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:12:07.361193 26184 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:12:07.361171 26185 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:12:07.361153 26187 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:12:07.362383 26179 server_base.cc:1034] running on GCE node
I20260926 15:12:07.363070 26179 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:12:07.364341 26179 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:12:07.365597 26179 hybrid_clock.cc:648] HybridClock initialized: now 1790435527365544 us; error 37 us; skew 500 ppm
I20260926 15:12:07.365952 26179 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:12:07 dist-test-slave-865v krb5kdc[26166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435527, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:12:07.369405 26179 init.cc:377] Logged in from keytab as kudu/127.12.34.254@KRBTEST.COM (short username kudu)
I20260926 15:12:07.370806 26179 webserver.cc:460] Webserver started at http://127.12.34.254:37331/ using document root <none> and password file <none>
I20260926 15:12:07.371472 26179 fs_manager.cc:359] Metadata directory not provided
I20260926 15:12:07.371595 26179 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:12:07.371886 26179 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:12:07.374866 26179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790435318278681-12427-0/minicluster-data/master-0/data/instance:
uuid: "7237447a1fd44b7ebb7fd2da4a0b2fa8"
format_stamp: "Formatted at 2026-09-26 15:12:07 on dist-test-slave-865v"
server_key: "48bbd0b7dcb8707d1d72a33f604bd9a6"
server_key_iv: "ecd27677d84f335f6634e1625f6aad75"
server_key_version: "encryptionkey@0"
I20260926 15:12:07.375574 26179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790435318278681-12427-0/minicluster-data/master-0/wal/instance:
uuid: "7237447a1fd44b7ebb7fd2da4a0b2fa8"
format_stamp: "Formatted at 2026-09-26 15:12:07 on dist-test-slave-865v"
server_key: "48bbd0b7dcb8707d1d72a33f604bd9a6"
server_key_iv: "ecd27677d84f335f6634e1625f6aad75"
server_key_version: "encryptionkey@0"
I20260926 15:12:07.381487 26179 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260926 15:12:07.384850 26194 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:12:07.386226 26179 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 15:12:07.386407 26179 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790435318278681-12427-0/minicluster-data/master-0/data,/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790435318278681-12427-0/minicluster-data/master-0/wal
uuid: "7237447a1fd44b7ebb7fd2da4a0b2fa8"
format_stamp: "Formatted at 2026-09-26 15:12:07 on dist-test-slave-865v"
server_key: "48bbd0b7dcb8707d1d72a33f604bd9a6"
server_key_iv: "ecd27677d84f335f6634e1625f6aad75"
server_key_version: "encryptionkey@0"
I20260926 15:12:07.386601 26179 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790435318278681-12427-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790435318278681-12427-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790435318278681-12427-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:12:07.413035 26179 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:12:07.417273 26179 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:12:07.417560 26179 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:12:07.437258 26245 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.34.254:46751 every 8 connection(s)
I20260926 15:12:07.437263 26179 rpc_server.cc:307] RPC server started. Bound to: 127.12.34.254:46751
I20260926 15:12:07.438745 26179 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
I20260926 15:12:07.442457 26246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:12:07.445611 12427 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8uoJGv/build/asan/bin/kudu as pid 26179
I20260926 15:12:07.445832 12427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790435318278681-12427-0/minicluster-data/master-0/wal/instance
I20260926 15:12:07.446650 12427 external_mini_cluster.cc:1428] Setting key 6291fa9df6925a57375889154a61f38c
I20260926 15:12:07.455440 26246 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7237447a1fd44b7ebb7fd2da4a0b2fa8: Bootstrap starting.
I20260926 15:12:07.459522 26246 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7237447a1fd44b7ebb7fd2da4a0b2fa8: Neither blocks nor log segments found. Creating new log.
I20260926 15:12:07.460958 26246 log.cc:824] T 00000000000000000000000000000000 P 7237447a1fd44b7ebb7fd2da4a0b2fa8: Log is configured to *not* fsync() on all Append() calls
Sep 26 15:12:07 dist-test-slave-865v krb5kdc[26166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435526, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.34.254@KRBTEST.COM
I20260926 15:12:07.464159 26246 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7237447a1fd44b7ebb7fd2da4a0b2fa8: No bootstrap required, opened a new log
I20260926 15:12:07.471794 26249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:07.448623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56400 (local address 127.12.34.254:46751)
0926 15:12:07.449432 (+ 809us) server_negotiation.cc:206] Beginning negotiation
0926 15:12:07.449443 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 15:12:07.449482 (+ 39us) server_negotiation.cc:407] Connection header received
0926 15:12:07.451113 (+ 1631us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:12:07.451134 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:12:07.451531 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:12:07.451832 (+ 301us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:12:07.453709 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:12:07.455031 (+ 1322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:12:07.457047 (+ 2016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:12:07.457777 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:12:07.457948 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:12:07.463609 (+ 5661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:12:07.463618 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:12:07.463633 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:12:07.463670 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:12:07.467336 (+ 3666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:12:07.468199 (+ 863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:12:07.468206 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:12:07.468216 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:12:07.468414 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:12:07.468859 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:12:07.468864 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:12:07.468866 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:12:07.469488 (+ 622us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:12:07.469831 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0926 15:12:07.470287 (+ 456us) server_negotiation.cc:299] Negotiation successful
0926 15:12:07.470759 (+ 472us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":612,"thread_start_us":415,"threads_started":1}
I20260926 15:12:07.472370 26246 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7237447a1fd44b7ebb7fd2da4a0b2fa8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7237447a1fd44b7ebb7fd2da4a0b2fa8" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 46751 } }
I20260926 15:12:07.472746 26246 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7237447a1fd44b7ebb7fd2da4a0b2fa8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:12:07.472862 26246 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7237447a1fd44b7ebb7fd2da4a0b2fa8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7237447a1fd44b7ebb7fd2da4a0b2fa8, State: Initialized, Role: FOLLOWER
I20260926 15:12:07.473515 26246 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7237447a1fd44b7ebb7fd2da4a0b2fa8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7237447a1fd44b7ebb7fd2da4a0b2fa8" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 46751 } }
I20260926 15:12:07.473752 26246 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7237447a1fd44b7ebb7fd2da4a0b2fa8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 15:12:07.473884 26246 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7237447a1fd44b7ebb7fd2da4a0b2fa8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 15:12:07.474058 26246 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7237447a1fd44b7ebb7fd2da4a0b2fa8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:12:07.475920 12427 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 15:12:07.476599 26246 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7237447a1fd44b7ebb7fd2da4a0b2fa8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7237447a1fd44b7ebb7fd2da4a0b2fa8" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 46751 } }
I20260926 15:12:07.477126 26246 leader_election.cc:302] T 00000000000000000000000000000000 P 7237447a1fd44b7ebb7fd2da4a0b2fa8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7237447a1fd44b7ebb7fd2da4a0b2fa8; no voters:
I20260926 15:12:07.477777 26246 leader_election.cc:288] T 00000000000000000000000000000000 P 7237447a1fd44b7ebb7fd2da4a0b2fa8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 15:12:07.477890 26251 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7237447a1fd44b7ebb7fd2da4a0b2fa8 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:12:07.480288 26246 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7237447a1fd44b7ebb7fd2da4a0b2fa8 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:12:07.481382 26251 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7237447a1fd44b7ebb7fd2da4a0b2fa8 [term 1 LEADER]: Becoming Leader. State: Replica: 7237447a1fd44b7ebb7fd2da4a0b2fa8, State: Running, Role: LEADER
I20260926 15:12:07.482007 26251 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7237447a1fd44b7ebb7fd2da4a0b2fa8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7237447a1fd44b7ebb7fd2da4a0b2fa8" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 46751 } }
I20260926 15:12:07.485544 26252 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7237447a1fd44b7ebb7fd2da4a0b2fa8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7237447a1fd44b7ebb7fd2da4a0b2fa8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7237447a1fd44b7ebb7fd2da4a0b2fa8" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 46751 } } }
I20260926 15:12:07.486308 26252 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7237447a1fd44b7ebb7fd2da4a0b2fa8 [sys.catalog]: This master's current role is: LEADER
I20260926 15:12:07.485556 26253 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7237447a1fd44b7ebb7fd2da4a0b2fa8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7237447a1fd44b7ebb7fd2da4a0b2fa8. Latest consensus state: current_term: 1 leader_uuid: "7237447a1fd44b7ebb7fd2da4a0b2fa8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7237447a1fd44b7ebb7fd2da4a0b2fa8" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 46751 } } }
I20260926 15:12:07.489323 26253 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7237447a1fd44b7ebb7fd2da4a0b2fa8 [sys.catalog]: This master's current role is: LEADER
I20260926 15:12:07.490149 26259 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:12:07.495532 26259 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 15:12:07.504740 26259 catalog_manager.cc:1347] Generated new cluster ID: 8ba57f004d614d0ab103d145989f2ee5
I20260926 15:12:07.504901 26259 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 15:12:07.514839 26259 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 15:12:07.516184 26259 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:12:07.526582 26259 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7237447a1fd44b7ebb7fd2da4a0b2fa8: Generated new TSK 0
I20260926 15:12:07.527329 26259 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 15:12:07.536376 12427 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 26179
I20260926 15:12:07.537578 26241 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:12:07.628796 12427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8uoJGv/build/asan/bin/kudu with pid 26179
2026-09-26T15:12:07Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1984 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 15:12:07 dist-test-slave-865v krb5kdc[26274](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 15:12:07 dist-test-slave-865v krb5kdc[26274](info): set up 2 sockets
Sep 26 15:12:07 dist-test-slave-865v krb5kdc[26274](info): commencing operation
krb5kdc: starting...
W20260926 15:12:08.749658 12427 mini_kdc.cc:121] Time spent starting KDC: real 1.084s user 0.002s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/krb5kdc/test-user.keytab.
Sep 26 15:12:08 dist-test-slave-865v krb5kdc[26274](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435528, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T15:12:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T15:12:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.34.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.34.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.34.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.254.keytab.
Entry for principal kudu/127.12.34.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.254.keytab.
I20260926 15:12:08.973017 12427 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8uoJGv/build/asan/bin/kudu
/tmp/dist-test-task8uoJGv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.34.254:43645
--webserver_interface=127.12.34.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.254.keytab
--principal=kudu/127.12.34.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.34.212:36007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.34.254:43645
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:12:09.284780 26287 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:12:09.285140 26287 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:12:09.285310 26287 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:12:09.292304 26287 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:12:09.292443 26287 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:12:09.292551 26287 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:12:09.292657 26287 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:12:09.292727 26287 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:12:09.302345 26287 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.34.212:36007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.34.254:43645
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.254.keytab
--principal=kudu/127.12.34.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.34.254:43645
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.34.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:12:09.303705 26287 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:12:09.304963 26287 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:12:09.316612 26295 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:12:09.316660 26293 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:12:09.316570 26292 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:12:09.317821 26287 server_base.cc:1034] running on GCE node
I20260926 15:12:09.318550 26287 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:12:09.319919 26287 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:12:09.321180 26287 hybrid_clock.cc:648] HybridClock initialized: now 1790435529321129 us; error 41 us; skew 500 ppm
I20260926 15:12:09.321525 26287 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:12:09 dist-test-slave-865v krb5kdc[26274](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435529, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:12:09.324584 26287 init.cc:377] Logged in from keytab as kudu/127.12.34.254@KRBTEST.COM (short username kudu)
I20260926 15:12:09.326050 26287 webserver.cc:460] Webserver started at http://127.12.34.254:42385/ using document root <none> and password file <none>
I20260926 15:12:09.326709 26287 fs_manager.cc:359] Metadata directory not provided
I20260926 15:12:09.326872 26287 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:12:09.327184 26287 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:12:09.330106 26287 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/master-0/data/instance:
uuid: "2e76b19894564aa28e81f161acd96f87"
format_stamp: "Formatted at 2026-09-26 15:12:09 on dist-test-slave-865v"
server_key: "4cd98550f90faf5dce91d40b3617c5c3"
server_key_iv: "5bc885476fa2d48422d0345de0ed7477"
server_key_version: "encryptionkey@0"
I20260926 15:12:09.330821 26287 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/master-0/wal/instance:
uuid: "2e76b19894564aa28e81f161acd96f87"
format_stamp: "Formatted at 2026-09-26 15:12:09 on dist-test-slave-865v"
server_key: "4cd98550f90faf5dce91d40b3617c5c3"
server_key_iv: "5bc885476fa2d48422d0345de0ed7477"
server_key_version: "encryptionkey@0"
I20260926 15:12:09.335703 26287 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 15:12:09.339022 26302 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:12:09.340258 26287 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260926 15:12:09.340437 26287 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/master-0/data,/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/master-0/wal
uuid: "2e76b19894564aa28e81f161acd96f87"
format_stamp: "Formatted at 2026-09-26 15:12:09 on dist-test-slave-865v"
server_key: "4cd98550f90faf5dce91d40b3617c5c3"
server_key_iv: "5bc885476fa2d48422d0345de0ed7477"
server_key_version: "encryptionkey@0"
I20260926 15:12:09.340636 26287 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:12:09.355214 26287 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:12:09.358986 26287 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:12:09.359279 26287 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:12:09.377974 26287 rpc_server.cc:307] RPC server started. Bound to: 127.12.34.254:43645
I20260926 15:12:09.378064 26353 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.34.254:43645 every 8 connection(s)
I20260926 15:12:09.379336 26287 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
I20260926 15:12:09.382925 26354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:12:09.383602 12427 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8uoJGv/build/asan/bin/kudu as pid 26287
I20260926 15:12:09.383831 12427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/master-0/wal/instance
I20260926 15:12:09.384497 12427 external_mini_cluster.cc:1428] Setting key 66f3af7ad3258577e4bbfe211c3defe9
I20260926 15:12:09.397226 26354 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e76b19894564aa28e81f161acd96f87: Bootstrap starting.
Sep 26 15:12:09 dist-test-slave-865v krb5kdc[26274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435528, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.34.254@KRBTEST.COM
I20260926 15:12:09.400369 26354 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2e76b19894564aa28e81f161acd96f87: Neither blocks nor log segments found. Creating new log.
I20260926 15:12:09.401544 26354 log.cc:824] T 00000000000000000000000000000000 P 2e76b19894564aa28e81f161acd96f87: Log is configured to *not* fsync() on all Append() calls
I20260926 15:12:09.405012 26354 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e76b19894564aa28e81f161acd96f87: No bootstrap required, opened a new log
I20260926 15:12:09.409477 26357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:09.386460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44646 (local address 127.12.34.254:43645)
0926 15:12:09.387251 (+ 791us) server_negotiation.cc:206] Beginning negotiation
0926 15:12:09.387265 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 15:12:09.387319 (+ 54us) server_negotiation.cc:407] Connection header received
0926 15:12:09.389317 (+ 1998us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:12:09.389342 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:12:09.389804 (+ 462us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:12:09.390236 (+ 432us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:12:09.392073 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:12:09.394080 (+ 2007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:12:09.395390 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:12:09.396383 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:12:09.396555 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:12:09.401145 (+ 4590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:12:09.401156 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:12:09.401177 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:12:09.401228 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:12:09.404989 (+ 3761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:12:09.405742 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:12:09.405750 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:12:09.405759 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:12:09.405960 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:12:09.406297 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:12:09.406302 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:12:09.406306 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:12:09.407016 (+ 710us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:12:09.407335 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0926 15:12:09.407696 (+ 361us) server_negotiation.cc:299] Negotiation successful
0926 15:12:09.408157 (+ 461us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":535,"thread_start_us":335,"threads_started":1}
I20260926 15:12:09.414472 26354 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2e76b19894564aa28e81f161acd96f87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e76b19894564aa28e81f161acd96f87" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 43645 } }
I20260926 15:12:09.414821 26354 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2e76b19894564aa28e81f161acd96f87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:12:09.414961 26354 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2e76b19894564aa28e81f161acd96f87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e76b19894564aa28e81f161acd96f87, State: Initialized, Role: FOLLOWER
I20260926 15:12:09.415580 26354 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2e76b19894564aa28e81f161acd96f87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e76b19894564aa28e81f161acd96f87" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 43645 } }
I20260926 15:12:09.415815 26354 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2e76b19894564aa28e81f161acd96f87 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 15:12:09.415951 26354 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2e76b19894564aa28e81f161acd96f87 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 15:12:09.416081 26354 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2e76b19894564aa28e81f161acd96f87 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:12:09.418496 26354 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2e76b19894564aa28e81f161acd96f87 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e76b19894564aa28e81f161acd96f87" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 43645 } }
I20260926 15:12:09.418936 26354 leader_election.cc:302] T 00000000000000000000000000000000 P 2e76b19894564aa28e81f161acd96f87 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2e76b19894564aa28e81f161acd96f87; no voters:
I20260926 15:12:09.419445 26354 leader_election.cc:288] T 00000000000000000000000000000000 P 2e76b19894564aa28e81f161acd96f87 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 15:12:09.419582 26359 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2e76b19894564aa28e81f161acd96f87 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:12:09.420255 26359 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2e76b19894564aa28e81f161acd96f87 [term 1 LEADER]: Becoming Leader. State: Replica: 2e76b19894564aa28e81f161acd96f87, State: Running, Role: LEADER
I20260926 15:12:09.420753 26359 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2e76b19894564aa28e81f161acd96f87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e76b19894564aa28e81f161acd96f87" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 43645 } }
I20260926 15:12:09.421430 26354 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2e76b19894564aa28e81f161acd96f87 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:12:09.423585 26361 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e76b19894564aa28e81f161acd96f87 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2e76b19894564aa28e81f161acd96f87. Latest consensus state: current_term: 1 leader_uuid: "2e76b19894564aa28e81f161acd96f87" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e76b19894564aa28e81f161acd96f87" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 43645 } } }
I20260926 15:12:09.424032 26361 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e76b19894564aa28e81f161acd96f87 [sys.catalog]: This master's current role is: LEADER
I20260926 15:12:09.424520 26360 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e76b19894564aa28e81f161acd96f87 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2e76b19894564aa28e81f161acd96f87" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e76b19894564aa28e81f161acd96f87" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 43645 } } }
I20260926 15:12:09.424824 26360 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e76b19894564aa28e81f161acd96f87 [sys.catalog]: This master's current role is: LEADER
I20260926 15:12:09.426110 26368 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:12:09.430341 26368 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 15:12:09.437796 26368 catalog_manager.cc:1347] Generated new cluster ID: 5b907a4a881e41c48bab7c4645ef1315
I20260926 15:12:09.437906 26368 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 15:12:09.459288 26368 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 15:12:09.460574 26368 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:12:09.467718 26368 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2e76b19894564aa28e81f161acd96f87: Generated new TSK 0
I20260926 15:12:09.468494 26368 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.34.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.34.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.34.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.193.keytab.
Entry for principal kudu/127.12.34.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.193.keytab.
I20260926 15:12:09.520685 12427 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8uoJGv/build/asan/bin/kudu
/tmp/dist-test-task8uoJGv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.34.193:0
--local_ip_for_outbound_sockets=127.12.34.193
--webserver_interface=127.12.34.193
--webserver_port=0
--tserver_master_addrs=127.12.34.254:43645
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.193.keytab
--principal=kudu/127.12.34.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.34.212:36007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:12:09.842731 26380 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:12:09.842989 26380 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:12:09.843146 26380 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:12:09.850580 26380 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:12:09.850680 26380 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:12:09.850870 26380 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.34.193
I20260926 15:12:09.860319 26380 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.34.212:36007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.193.keytab
--principal=kudu/127.12.34.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.34.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.34.193
--webserver_port=0
--tserver_master_addrs=127.12.34.254:43645
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.34.193
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:12:09.861594 26380 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:12:09.862870 26380 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:12:09.876019 26385 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:12:09.876004 26388 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:12:09.876161 26386 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:12:09.877390 26380 server_base.cc:1034] running on GCE node
I20260926 15:12:09.878098 26380 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:12:09.878985 26380 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:12:09.880211 26380 hybrid_clock.cc:648] HybridClock initialized: now 1790435529880159 us; error 47 us; skew 500 ppm
I20260926 15:12:09.880517 26380 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:12:09 dist-test-slave-865v krb5kdc[26274](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435529, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:12:09.883785 26380 init.cc:377] Logged in from keytab as kudu/127.12.34.193@KRBTEST.COM (short username kudu)
I20260926 15:12:09.885227 26380 webserver.cc:460] Webserver started at http://127.12.34.193:33943/ using document root <none> and password file <none>
I20260926 15:12:09.885870 26380 fs_manager.cc:359] Metadata directory not provided
I20260926 15:12:09.886004 26380 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:12:09.886291 26380 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:12:09.889045 26380 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/ts-0/data/instance:
uuid: "4067cd15627b4f5ba7f5eb07551e7fab"
format_stamp: "Formatted at 2026-09-26 15:12:09 on dist-test-slave-865v"
server_key: "26f9593fa47ac2148c4a20256e4c7f86"
server_key_iv: "2c7606219edfd0d3e9eea39aa26ad591"
server_key_version: "encryptionkey@0"
I20260926 15:12:09.889843 26380 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/ts-0/wal/instance:
uuid: "4067cd15627b4f5ba7f5eb07551e7fab"
format_stamp: "Formatted at 2026-09-26 15:12:09 on dist-test-slave-865v"
server_key: "26f9593fa47ac2148c4a20256e4c7f86"
server_key_iv: "2c7606219edfd0d3e9eea39aa26ad591"
server_key_version: "encryptionkey@0"
I20260926 15:12:09.894626 26380 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 15:12:09.898273 26395 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:12:09.899756 26380 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 15:12:09.899931 26380 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/ts-0/data,/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/ts-0/wal
uuid: "4067cd15627b4f5ba7f5eb07551e7fab"
format_stamp: "Formatted at 2026-09-26 15:12:09 on dist-test-slave-865v"
server_key: "26f9593fa47ac2148c4a20256e4c7f86"
server_key_iv: "2c7606219edfd0d3e9eea39aa26ad591"
server_key_version: "encryptionkey@0"
I20260926 15:12:09.900130 26380 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:12:09.918263 26380 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:12:09.922744 26380 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:12:09.923027 26380 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:12:09.923971 26380 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:12:09.925412 26380 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:12:09.925515 26380 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:12:09.925657 26380 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:12:09.925822 26380 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:12:09.961613 26380 rpc_server.cc:307] RPC server started. Bound to: 127.12.34.193:35363
I20260926 15:12:09.961661 26507 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.34.193:35363 every 8 connection(s)
I20260926 15:12:09.962894 26380 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/ts-0/data/info.pb
I20260926 15:12:09.972680 12427 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8uoJGv/build/asan/bin/kudu as pid 26380
I20260926 15:12:09.972892 12427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/ts-0/wal/instance
I20260926 15:12:09.973556 12427 external_mini_cluster.cc:1428] Setting key 0cd373158e50e83ea6600a0f446655ac
Sep 26 15:12:09 dist-test-slave-865v krb5kdc[26274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435529, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.193@KRBTEST.COM for kudu/127.12.34.254@KRBTEST.COM
I20260926 15:12:09.982854 26510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:09.965349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.193:46225 (local address 127.12.34.254:43645)
0926 15:12:09.965815 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0926 15:12:09.965820 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:12:09.966720 (+ 900us) server_negotiation.cc:407] Connection header received
0926 15:12:09.968636 (+ 1916us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:12:09.968642 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:12:09.968731 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:12:09.968849 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:12:09.970321 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:12:09.971034 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:12:09.972546 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:12:09.972982 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:12:09.973115 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:12:09.977279 (+ 4164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:12:09.977284 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:12:09.977287 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:12:09.977322 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:12:09.979702 (+ 2380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:12:09.980616 (+ 914us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:12:09.980620 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:12:09.980622 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:12:09.980731 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:12:09.981239 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:12:09.981242 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:12:09.981244 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:12:09.981495 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:12:09.981613 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0926 15:12:09.982424 (+ 811us) server_negotiation.cc:299] Negotiation successful
0926 15:12:09.982590 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":236,"threads_started":1}
I20260926 15:12:09.984009 26511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:09.965914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:43645 (local address 127.12.34.193:46225)
0926 15:12:09.966576 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0926 15:12:09.966611 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 15:12:09.968385 (+ 1774us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:12:09.969071 (+ 686us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:12:09.969122 (+ 51us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:12:09.969603 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:12:09.970159 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:12:09.970175 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:12:09.971183 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:12:09.971193 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:12:09.972372 (+ 1179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:12:09.972384 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:12:09.973219 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:12:09.973226 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:12:09.973439 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:12:09.974131 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:12:09.974157 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:12:09.977048 (+ 2891us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:12:09.979842 (+ 2794us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:12:09.979854 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:12:09.979871 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:12:09.980500 (+ 629us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:12:09.980845 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:12:09.980853 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:12:09.980859 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:12:09.981132 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:12:09.981619 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:12:09.981630 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:12:09.982051 (+ 421us) client_negotiation.cc:769] Sending connection context
0926 15:12:09.982338 (+ 287us) client_negotiation.cc:240] Negotiation successful
0926 15:12:09.982887 (+ 549us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":252,"threads_started":1}
I20260926 15:12:09.985564 26508 heartbeater.cc:347] Connected to a master server at 127.12.34.254:43645
I20260926 15:12:09.985862 26508 heartbeater.cc:464] Registering TS with master...
I20260926 15:12:09.986688 26508 heartbeater.cc:511] Master 127.12.34.254:43645 requested a full tablet report, sending...
I20260926 15:12:09.988807 26319 ts_manager.cc:194] Registered new tserver with Master: 4067cd15627b4f5ba7f5eb07551e7fab (127.12.34.193:35363)
I20260926 15:12:09.990448 26319 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.34.193@KRBTEST.COM'} at 127.12.34.193:46225
WARNING: no policy specified for kudu/127.12.34.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.34.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.34.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.194.keytab.
Entry for principal kudu/127.12.34.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.194.keytab.
I20260926 15:12:10.030076 12427 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8uoJGv/build/asan/bin/kudu
/tmp/dist-test-task8uoJGv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.34.194:0
--local_ip_for_outbound_sockets=127.12.34.194
--webserver_interface=127.12.34.194
--webserver_port=0
--tserver_master_addrs=127.12.34.254:43645
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.194.keytab
--principal=kudu/127.12.34.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.34.212:36007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:12:10.373593 26514 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:12:10.373862 26514 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:12:10.373963 26514 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:12:10.383422 26514 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:12:10.383566 26514 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:12:10.383729 26514 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.34.194
I20260926 15:12:10.399470 26514 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.34.212:36007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.194.keytab
--principal=kudu/127.12.34.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.34.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.34.194
--webserver_port=0
--tserver_master_addrs=127.12.34.254:43645
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.34.194
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:12:10.400961 26514 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:12:10.402381 26514 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:12:10.415588 26519 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:12:10.415637 26522 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:12:10.415658 26520 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:12:10.416584 26514 server_base.cc:1034] running on GCE node
I20260926 15:12:10.417256 26514 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:12:10.418260 26514 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:12:10.419493 26514 hybrid_clock.cc:648] HybridClock initialized: now 1790435530419436 us; error 53 us; skew 500 ppm
I20260926 15:12:10.419816 26514 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:12:10 dist-test-slave-865v krb5kdc[26274](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435530, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:12:10.423183 26514 init.cc:377] Logged in from keytab as kudu/127.12.34.194@KRBTEST.COM (short username kudu)
I20260926 15:12:10.424556 26514 webserver.cc:460] Webserver started at http://127.12.34.194:38851/ using document root <none> and password file <none>
I20260926 15:12:10.425282 26514 fs_manager.cc:359] Metadata directory not provided
I20260926 15:12:10.425386 26514 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:12:10.425680 26514 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:12:10.428493 26514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/ts-1/data/instance:
uuid: "cbe51bf32d864c85a4189dc5531617b1"
format_stamp: "Formatted at 2026-09-26 15:12:10 on dist-test-slave-865v"
server_key: "b57409c04ecb3bf07d4f7f77e12c6e23"
server_key_iv: "807e84f89a714d99727fafcd477a3854"
server_key_version: "encryptionkey@0"
I20260926 15:12:10.429327 26514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/ts-1/wal/instance:
uuid: "cbe51bf32d864c85a4189dc5531617b1"
format_stamp: "Formatted at 2026-09-26 15:12:10 on dist-test-slave-865v"
server_key: "b57409c04ecb3bf07d4f7f77e12c6e23"
server_key_iv: "807e84f89a714d99727fafcd477a3854"
server_key_version: "encryptionkey@0"
I20260926 15:12:10.434329 26514 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 15:12:10.437655 26529 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:12:10.438954 26514 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 15:12:10.439163 26514 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/ts-1/data,/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/ts-1/wal
uuid: "cbe51bf32d864c85a4189dc5531617b1"
format_stamp: "Formatted at 2026-09-26 15:12:10 on dist-test-slave-865v"
server_key: "b57409c04ecb3bf07d4f7f77e12c6e23"
server_key_iv: "807e84f89a714d99727fafcd477a3854"
server_key_version: "encryptionkey@0"
I20260926 15:12:10.439364 26514 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:12:10.456784 26514 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:12:10.461004 26514 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:12:10.461309 26514 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:12:10.462235 26514 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:12:10.463584 26514 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:12:10.463685 26514 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:12:10.463830 26514 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:12:10.463912 26514 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:12:10.497355 26514 rpc_server.cc:307] RPC server started. Bound to: 127.12.34.194:37165
I20260926 15:12:10.497392 26641 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.34.194:37165 every 8 connection(s)
I20260926 15:12:10.498667 26514 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/ts-1/data/info.pb
I20260926 15:12:10.507220 12427 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8uoJGv/build/asan/bin/kudu as pid 26514
I20260926 15:12:10.507586 12427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/ts-1/wal/instance
I20260926 15:12:10.508213 12427 external_mini_cluster.cc:1428] Setting key 9f5e23ea64e111da5765555dcb064409
Sep 26 15:12:10 dist-test-slave-865v krb5kdc[26274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435530, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.194@KRBTEST.COM for kudu/127.12.34.254@KRBTEST.COM
I20260926 15:12:10.523821 26644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:10.501307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.194:54849 (local address 127.12.34.254:43645)
0926 15:12:10.501767 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0926 15:12:10.501773 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:12:10.502798 (+ 1025us) server_negotiation.cc:407] Connection header received
0926 15:12:10.504591 (+ 1793us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:12:10.504595 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:12:10.504662 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:12:10.504758 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:12:10.506231 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:12:10.506968 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:12:10.509559 (+ 2591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:12:10.510084 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:12:10.510267 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:12:10.515391 (+ 5124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:12:10.515396 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:12:10.515400 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:12:10.515441 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:12:10.519544 (+ 4103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:12:10.520646 (+ 1102us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:12:10.520653 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:12:10.520655 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:12:10.520826 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:12:10.521406 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:12:10.521410 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:12:10.521412 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:12:10.521753 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:12:10.521917 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0926 15:12:10.523297 (+ 1380us) server_negotiation.cc:299] Negotiation successful
0926 15:12:10.523507 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":198,"threads_started":1}
I20260926 15:12:10.526100 26645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:10.501941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:43645 (local address 127.12.34.194:54849)
0926 15:12:10.502636 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0926 15:12:10.502675 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 15:12:10.504276 (+ 1601us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:12:10.504952 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:12:10.504959 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:12:10.505514 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:12:10.506034 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:12:10.506050 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:12:10.507175 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:12:10.507179 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:12:10.509285 (+ 2106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:12:10.509314 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:12:10.510416 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:12:10.510426 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:12:10.510733 (+ 307us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:12:10.511744 (+ 1011us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:12:10.511783 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:12:10.515098 (+ 3315us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:12:10.519897 (+ 4799us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:12:10.519911 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:12:10.519926 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:12:10.520444 (+ 518us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:12:10.520961 (+ 517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:12:10.520966 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:12:10.520970 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:12:10.521272 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:12:10.521992 (+ 720us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:12:10.522007 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:12:10.522778 (+ 771us) client_negotiation.cc:769] Sending connection context
0926 15:12:10.523204 (+ 426us) client_negotiation.cc:240] Negotiation successful
0926 15:12:10.523958 (+ 754us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":279,"threads_started":1}
I20260926 15:12:10.528347 26642 heartbeater.cc:347] Connected to a master server at 127.12.34.254:43645
I20260926 15:12:10.528690 26642 heartbeater.cc:464] Registering TS with master...
I20260926 15:12:10.529551 26642 heartbeater.cc:511] Master 127.12.34.254:43645 requested a full tablet report, sending...
I20260926 15:12:10.531363 26319 ts_manager.cc:194] Registered new tserver with Master: cbe51bf32d864c85a4189dc5531617b1 (127.12.34.194:37165)
I20260926 15:12:10.532256 26319 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.34.194@KRBTEST.COM'} at 127.12.34.194:54849
WARNING: no policy specified for kudu/127.12.34.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.34.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.34.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.195.keytab.
Entry for principal kudu/127.12.34.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.195.keytab.
I20260926 15:12:10.570580 12427 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8uoJGv/build/asan/bin/kudu
/tmp/dist-test-task8uoJGv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.34.195:0
--local_ip_for_outbound_sockets=127.12.34.195
--webserver_interface=127.12.34.195
--webserver_port=0
--tserver_master_addrs=127.12.34.254:43645
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.195.keytab
--principal=kudu/127.12.34.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.34.212:36007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:12:10.895316 26648 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:12:10.895593 26648 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:12:10.895686 26648 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:12:10.902812 26648 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:12:10.902920 26648 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:12:10.903031 26648 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.34.195
I20260926 15:12:10.912747 26648 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.34.212:36007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.195.keytab
--principal=kudu/127.12.34.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.34.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.34.195
--webserver_port=0
--tserver_master_addrs=127.12.34.254:43645
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.34.195
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:12:10.913973 26648 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:12:10.915336 26648 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:12:10.928565 26656 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:12:10.928512 26653 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:12:10.928521 26654 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:12:10.929289 26648 server_base.cc:1034] running on GCE node
I20260926 15:12:10.929937 26648 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:12:10.930874 26648 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:12:10.932121 26648 hybrid_clock.cc:648] HybridClock initialized: now 1790435530932087 us; error 64 us; skew 500 ppm
I20260926 15:12:10.932767 26648 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:12:10 dist-test-slave-865v krb5kdc[26274](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435530, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:12:10.936614 26648 init.cc:377] Logged in from keytab as kudu/127.12.34.195@KRBTEST.COM (short username kudu)
I20260926 15:12:10.938442 26648 webserver.cc:460] Webserver started at http://127.12.34.195:35437/ using document root <none> and password file <none>
I20260926 15:12:10.939177 26648 fs_manager.cc:359] Metadata directory not provided
I20260926 15:12:10.939332 26648 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:12:10.939640 26648 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:12:10.942730 26648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/ts-2/data/instance:
uuid: "b440b82da2b24d09ac47aceedd225013"
format_stamp: "Formatted at 2026-09-26 15:12:10 on dist-test-slave-865v"
server_key: "12aec6bcbdef59bc94e5bbd9e3b0a73b"
server_key_iv: "5a9e03d0524afb3807e1879b21bc934c"
server_key_version: "encryptionkey@0"
I20260926 15:12:10.943593 26648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/ts-2/wal/instance:
uuid: "b440b82da2b24d09ac47aceedd225013"
format_stamp: "Formatted at 2026-09-26 15:12:10 on dist-test-slave-865v"
server_key: "12aec6bcbdef59bc94e5bbd9e3b0a73b"
server_key_iv: "5a9e03d0524afb3807e1879b21bc934c"
server_key_version: "encryptionkey@0"
I20260926 15:12:10.949060 26648 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260926 15:12:10.952580 26663 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:12:10.954316 26648 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 15:12:10.954483 26648 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/ts-2/data,/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/ts-2/wal
uuid: "b440b82da2b24d09ac47aceedd225013"
format_stamp: "Formatted at 2026-09-26 15:12:10 on dist-test-slave-865v"
server_key: "12aec6bcbdef59bc94e5bbd9e3b0a73b"
server_key_iv: "5a9e03d0524afb3807e1879b21bc934c"
server_key_version: "encryptionkey@0"
I20260926 15:12:10.954746 26648 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:12:10.979534 26648 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:12:10.984197 26648 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:12:10.984519 26648 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:12:10.985502 26648 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:12:10.986793 26648 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:12:10.986877 26648 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:12:10.987056 26648 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:12:10.987107 26648 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:12:10.993882 26508 heartbeater.cc:503] Master 127.12.34.254:43645 was elected leader, sending a full tablet report...
I20260926 15:12:11.018714 26648 rpc_server.cc:307] RPC server started. Bound to: 127.12.34.195:35357
I20260926 15:12:11.018755 26775 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.34.195:35357 every 8 connection(s)
I20260926 15:12:11.020021 26648 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/ts-2/data/info.pb
I20260926 15:12:11.024410 12427 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8uoJGv/build/asan/bin/kudu as pid 26648
I20260926 15:12:11.024605 12427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/ts-2/wal/instance
I20260926 15:12:11.025137 12427 external_mini_cluster.cc:1428] Setting key 3884ec9697c57396becf91f3c99a8d11
Sep 26 15:12:11 dist-test-slave-865v krb5kdc[26274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435530, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.195@KRBTEST.COM for kudu/127.12.34.254@KRBTEST.COM
I20260926 15:12:11.040481 26644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:11.022576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.195:37329 (local address 127.12.34.254:43645)
0926 15:12:11.022783 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0926 15:12:11.022789 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:12:11.023983 (+ 1194us) server_negotiation.cc:407] Connection header received
0926 15:12:11.026520 (+ 2537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:12:11.026525 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:12:11.026634 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:12:11.026743 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:12:11.028522 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:12:11.029350 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:12:11.031140 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:12:11.031524 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:12:11.031655 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:12:11.034913 (+ 3258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:12:11.034918 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:12:11.034921 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:12:11.034966 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:12:11.037622 (+ 2656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:12:11.038247 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:12:11.038250 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:12:11.038252 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:12:11.038360 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:12:11.038775 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:12:11.038779 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:12:11.038782 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:12:11.039082 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:12:11.039268 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0926 15:12:11.039904 (+ 636us) server_negotiation.cc:299] Negotiation successful
0926 15:12:11.040136 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 15:12:11.041460 26778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:11.023072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:43645 (local address 127.12.34.195:37329)
0926 15:12:11.023825 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0926 15:12:11.023867 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 15:12:11.026074 (+ 2207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:12:11.027024 (+ 950us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:12:11.027033 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:12:11.027701 (+ 668us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:12:11.028339 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:12:11.028355 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:12:11.029496 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:12:11.029501 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:12:11.031003 (+ 1502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:12:11.031015 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:12:11.031741 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:12:11.031748 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:12:11.031978 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:12:11.032576 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:12:11.032602 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:12:11.034727 (+ 2125us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:12:11.037737 (+ 3010us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:12:11.037743 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:12:11.037753 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:12:11.038152 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:12:11.038471 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:12:11.038475 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:12:11.038477 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:12:11.038661 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:12:11.039211 (+ 550us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:12:11.039219 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:12:11.039534 (+ 315us) client_negotiation.cc:769] Sending connection context
0926 15:12:11.039825 (+ 291us) client_negotiation.cc:240] Negotiation successful
0926 15:12:11.040312 (+ 487us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":338,"threads_started":1}
I20260926 15:12:11.042824 26776 heartbeater.cc:347] Connected to a master server at 127.12.34.254:43645
I20260926 15:12:11.043093 26776 heartbeater.cc:464] Registering TS with master...
I20260926 15:12:11.043919 26776 heartbeater.cc:511] Master 127.12.34.254:43645 requested a full tablet report, sending...
I20260926 15:12:11.045300 26319 ts_manager.cc:194] Registered new tserver with Master: b440b82da2b24d09ac47aceedd225013 (127.12.34.195:35357)
I20260926 15:12:11.046092 26319 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.34.195@KRBTEST.COM'} at 127.12.34.195:37329
I20260926 15:12:11.053153 12427 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 15:12:11.069921 26644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:11.057542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44654 (local address 127.12.34.254:43645)
0926 15:12:11.057720 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0926 15:12:11.057724 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:12:11.058166 (+ 442us) server_negotiation.cc:407] Connection header received
0926 15:12:11.058299 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:12:11.058302 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:12:11.058375 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:12:11.058456 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:12:11.059853 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:12:11.060648 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:12:11.062037 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:12:11.062471 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:12:11.062577 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:12:11.064931 (+ 2354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:12:11.064936 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:12:11.064939 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:12:11.064969 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:12:11.067611 (+ 2642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:12:11.068271 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:12:11.068275 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:12:11.068277 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:12:11.068402 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:12:11.068706 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:12:11.068708 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:12:11.068711 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:12:11.068981 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:12:11.069236 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0926 15:12:11.069531 (+ 295us) server_negotiation.cc:299] Negotiation successful
0926 15:12:11.069677 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 15:12:11.073935 26319 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44654:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 15:12:11.080565 26319 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 15:12:11.103348 26790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:11.092295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43732 (local address 127.12.34.193:35363)
0926 15:12:11.094069 (+ 1774us) server_negotiation.cc:206] Beginning negotiation
0926 15:12:11.094078 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 15:12:11.094641 (+ 563us) server_negotiation.cc:407] Connection header received
0926 15:12:11.095037 (+ 396us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:12:11.095044 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:12:11.095269 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:12:11.095425 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:12:11.096470 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:12:11.098131 (+ 1661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:12:11.100640 (+ 2509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:12:11.102139 (+ 1499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:12:11.102292 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:12:11.102531 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0926 15:12:11.102802 (+ 271us) server_negotiation.cc:299] Negotiation successful
0926 15:12:11.102963 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1610,"thread_start_us":1475,"threads_started":1}
I20260926 15:12:11.103426 26789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:11.092003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.193:35363 (local address 127.0.0.1:43732)
0926 15:12:11.094496 (+ 2493us) negotiation.cc:107] Waiting for socket to connect
0926 15:12:11.094534 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 15:12:11.094795 (+ 261us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:12:11.095744 (+ 949us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:12:11.095754 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:12:11.095782 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:12:11.096262 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:12:11.096274 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:12:11.098309 (+ 2035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:12:11.098314 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:12:11.100480 (+ 2166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:12:11.100494 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:12:11.102328 (+ 1834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:12:11.102334 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:12:11.102556 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:12:11.102608 (+ 52us) client_negotiation.cc:769] Sending connection context
0926 15:12:11.102768 (+ 160us) client_negotiation.cc:240] Negotiation successful
0926 15:12:11.102968 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2227,"spinlock_wait_cycles":7168,"thread_start_us":2261,"threads_started":1}
I20260926 15:12:11.106289 26791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:11.092796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57348 (local address 127.12.34.194:37165)
0926 15:12:11.093587 (+ 791us) server_negotiation.cc:206] Beginning negotiation
0926 15:12:11.093599 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 15:12:11.094640 (+ 1041us) server_negotiation.cc:407] Connection header received
0926 15:12:11.095036 (+ 396us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:12:11.095045 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:12:11.095269 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:12:11.095425 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:12:11.097332 (+ 1907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:12:11.098803 (+ 1471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:12:11.103353 (+ 4550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:12:11.104800 (+ 1447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:12:11.104926 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:12:11.105150 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0926 15:12:11.105795 (+ 645us) server_negotiation.cc:299] Negotiation successful
0926 15:12:11.105975 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":600,"thread_start_us":475,"threads_started":1}
I20260926 15:12:11.106334 26788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:11.092220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.194:37165 (local address 127.0.0.1:57348)
0926 15:12:11.094502 (+ 2282us) negotiation.cc:107] Waiting for socket to connect
0926 15:12:11.094535 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0926 15:12:11.094795 (+ 260us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:12:11.096715 (+ 1920us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:12:11.096719 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:12:11.096744 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:12:11.097157 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:12:11.097169 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:12:11.098983 (+ 1814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:12:11.098988 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:12:11.103184 (+ 4196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:12:11.103198 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:12:11.105314 (+ 2116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:12:11.105320 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:12:11.105515 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:12:11.105615 (+ 100us) client_negotiation.cc:769] Sending connection context
0926 15:12:11.105710 (+ 95us) client_negotiation.cc:240] Negotiation successful
0926 15:12:11.105921 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2165,"mutex_wait_us":22,"spinlock_wait_cycles":5632,"thread_start_us":1860,"threads_started":1}
I20260926 15:12:11.107643 26443 tablet_service.cc:1467] Processing CreateTablet for tablet 4cade3193eac4883bce2f4599ffc7b68 (DEFAULT_TABLE table=test-table [id=201d197185dd48eda45cabcc6ae97d1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:12:11.109339 26443 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4cade3193eac4883bce2f4599ffc7b68. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:12:11.109432 26792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:11.094927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.195:35357 (local address 127.0.0.1:57602)
0926 15:12:11.101591 (+ 6664us) negotiation.cc:107] Waiting for socket to connect
0926 15:12:11.101614 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 15:12:11.101739 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:12:11.102578 (+ 839us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:12:11.102584 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:12:11.102608 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:12:11.103081 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:12:11.103095 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:12:11.105062 (+ 1967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:12:11.105067 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:12:11.106970 (+ 1903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:12:11.106982 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:12:11.108605 (+ 1623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:12:11.108612 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:12:11.108795 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:12:11.108840 (+ 45us) client_negotiation.cc:769] Sending connection context
0926 15:12:11.108928 (+ 88us) client_negotiation.cc:240] Negotiation successful
0926 15:12:11.109060 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6544,"mutex_wait_us":48,"thread_start_us":6395,"threads_started":1}
I20260926 15:12:11.109901 26793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:11.095205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57602 (local address 127.12.34.195:35357)
0926 15:12:11.101119 (+ 5914us) server_negotiation.cc:206] Beginning negotiation
0926 15:12:11.101127 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 15:12:11.101719 (+ 592us) server_negotiation.cc:407] Connection header received
0926 15:12:11.101908 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:12:11.101917 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:12:11.102146 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:12:11.102324 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:12:11.103354 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:12:11.104904 (+ 1550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:12:11.107116 (+ 2212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:12:11.108405 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:12:11.108531 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:12:11.108726 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0926 15:12:11.109457 (+ 731us) server_negotiation.cc:299] Negotiation successful
0926 15:12:11.109608 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5703,"thread_start_us":5555,"threads_started":1}
I20260926 15:12:11.109907 26577 tablet_service.cc:1467] Processing CreateTablet for tablet 4cade3193eac4883bce2f4599ffc7b68 (DEFAULT_TABLE table=test-table [id=201d197185dd48eda45cabcc6ae97d1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:12:11.111114 26577 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4cade3193eac4883bce2f4599ffc7b68. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:12:11.113179 26711 tablet_service.cc:1467] Processing CreateTablet for tablet 4cade3193eac4883bce2f4599ffc7b68 (DEFAULT_TABLE table=test-table [id=201d197185dd48eda45cabcc6ae97d1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:12:11.114423 26711 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4cade3193eac4883bce2f4599ffc7b68. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:12:11.123183 26794 tablet_bootstrap.cc:492] T 4cade3193eac4883bce2f4599ffc7b68 P cbe51bf32d864c85a4189dc5531617b1: Bootstrap starting.
I20260926 15:12:11.123963 26795 tablet_bootstrap.cc:492] T 4cade3193eac4883bce2f4599ffc7b68 P 4067cd15627b4f5ba7f5eb07551e7fab: Bootstrap starting.
I20260926 15:12:11.126924 26794 tablet_bootstrap.cc:654] T 4cade3193eac4883bce2f4599ffc7b68 P cbe51bf32d864c85a4189dc5531617b1: Neither blocks nor log segments found. Creating new log.
I20260926 15:12:11.127542 26795 tablet_bootstrap.cc:654] T 4cade3193eac4883bce2f4599ffc7b68 P 4067cd15627b4f5ba7f5eb07551e7fab: Neither blocks nor log segments found. Creating new log.
I20260926 15:12:11.128175 26796 tablet_bootstrap.cc:492] T 4cade3193eac4883bce2f4599ffc7b68 P b440b82da2b24d09ac47aceedd225013: Bootstrap starting.
I20260926 15:12:11.128213 26794 log.cc:824] T 4cade3193eac4883bce2f4599ffc7b68 P cbe51bf32d864c85a4189dc5531617b1: Log is configured to *not* fsync() on all Append() calls
I20260926 15:12:11.128753 26795 log.cc:824] T 4cade3193eac4883bce2f4599ffc7b68 P 4067cd15627b4f5ba7f5eb07551e7fab: Log is configured to *not* fsync() on all Append() calls
I20260926 15:12:11.131632 26796 tablet_bootstrap.cc:654] T 4cade3193eac4883bce2f4599ffc7b68 P b440b82da2b24d09ac47aceedd225013: Neither blocks nor log segments found. Creating new log.
I20260926 15:12:11.131671 26794 tablet_bootstrap.cc:492] T 4cade3193eac4883bce2f4599ffc7b68 P cbe51bf32d864c85a4189dc5531617b1: No bootstrap required, opened a new log
I20260926 15:12:11.131763 26795 tablet_bootstrap.cc:492] T 4cade3193eac4883bce2f4599ffc7b68 P 4067cd15627b4f5ba7f5eb07551e7fab: No bootstrap required, opened a new log
I20260926 15:12:11.131938 26794 ts_tablet_manager.cc:1397] T 4cade3193eac4883bce2f4599ffc7b68 P cbe51bf32d864c85a4189dc5531617b1: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.006s
I20260926 15:12:11.132077 26795 ts_tablet_manager.cc:1397] T 4cade3193eac4883bce2f4599ffc7b68 P 4067cd15627b4f5ba7f5eb07551e7fab: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260926 15:12:11.132865 26796 log.cc:824] T 4cade3193eac4883bce2f4599ffc7b68 P b440b82da2b24d09ac47aceedd225013: Log is configured to *not* fsync() on all Append() calls
I20260926 15:12:11.135692 26796 tablet_bootstrap.cc:492] T 4cade3193eac4883bce2f4599ffc7b68 P b440b82da2b24d09ac47aceedd225013: No bootstrap required, opened a new log
I20260926 15:12:11.135934 26796 ts_tablet_manager.cc:1397] T 4cade3193eac4883bce2f4599ffc7b68 P b440b82da2b24d09ac47aceedd225013: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260926 15:12:11.139240 26794 raft_consensus.cc:348] T 4cade3193eac4883bce2f4599ffc7b68 P cbe51bf32d864c85a4189dc5531617b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4067cd15627b4f5ba7f5eb07551e7fab" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 35363 } } peers { permanent_uuid: "cbe51bf32d864c85a4189dc5531617b1" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 37165 } } peers { permanent_uuid: "b440b82da2b24d09ac47aceedd225013" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 35357 } }
I20260926 15:12:11.139598 26794 raft_consensus.cc:374] T 4cade3193eac4883bce2f4599ffc7b68 P cbe51bf32d864c85a4189dc5531617b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:12:11.139724 26794 raft_consensus.cc:729] T 4cade3193eac4883bce2f4599ffc7b68 P cbe51bf32d864c85a4189dc5531617b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbe51bf32d864c85a4189dc5531617b1, State: Initialized, Role: FOLLOWER
I20260926 15:12:11.140301 26794 consensus_queue.cc:260] T 4cade3193eac4883bce2f4599ffc7b68 P cbe51bf32d864c85a4189dc5531617b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4067cd15627b4f5ba7f5eb07551e7fab" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 35363 } } peers { permanent_uuid: "cbe51bf32d864c85a4189dc5531617b1" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 37165 } } peers { permanent_uuid: "b440b82da2b24d09ac47aceedd225013" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 35357 } }
I20260926 15:12:11.142025 26794 ts_tablet_manager.cc:1428] T 4cade3193eac4883bce2f4599ffc7b68 P cbe51bf32d864c85a4189dc5531617b1: Time spent starting tablet: real 0.010s user 0.009s sys 0.003s
I20260926 15:12:11.142213 26795 raft_consensus.cc:348] T 4cade3193eac4883bce2f4599ffc7b68 P 4067cd15627b4f5ba7f5eb07551e7fab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4067cd15627b4f5ba7f5eb07551e7fab" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 35363 } } peers { permanent_uuid: "cbe51bf32d864c85a4189dc5531617b1" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 37165 } } peers { permanent_uuid: "b440b82da2b24d09ac47aceedd225013" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 35357 } }
I20260926 15:12:11.142613 26642 heartbeater.cc:503] Master 127.12.34.254:43645 was elected leader, sending a full tablet report...
I20260926 15:12:11.142678 26795 raft_consensus.cc:374] T 4cade3193eac4883bce2f4599ffc7b68 P 4067cd15627b4f5ba7f5eb07551e7fab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:12:11.142773 26795 raft_consensus.cc:729] T 4cade3193eac4883bce2f4599ffc7b68 P 4067cd15627b4f5ba7f5eb07551e7fab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4067cd15627b4f5ba7f5eb07551e7fab, State: Initialized, Role: FOLLOWER
I20260926 15:12:11.143337 26795 consensus_queue.cc:260] T 4cade3193eac4883bce2f4599ffc7b68 P 4067cd15627b4f5ba7f5eb07551e7fab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4067cd15627b4f5ba7f5eb07551e7fab" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 35363 } } peers { permanent_uuid: "cbe51bf32d864c85a4189dc5531617b1" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 37165 } } peers { permanent_uuid: "b440b82da2b24d09ac47aceedd225013" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 35357 } }
I20260926 15:12:11.145098 26795 ts_tablet_manager.cc:1428] T 4cade3193eac4883bce2f4599ffc7b68 P 4067cd15627b4f5ba7f5eb07551e7fab: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260926 15:12:11.146481 26796 raft_consensus.cc:348] T 4cade3193eac4883bce2f4599ffc7b68 P b440b82da2b24d09ac47aceedd225013 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4067cd15627b4f5ba7f5eb07551e7fab" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 35363 } } peers { permanent_uuid: "cbe51bf32d864c85a4189dc5531617b1" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 37165 } } peers { permanent_uuid: "b440b82da2b24d09ac47aceedd225013" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 35357 } }
I20260926 15:12:11.146936 26796 raft_consensus.cc:374] T 4cade3193eac4883bce2f4599ffc7b68 P b440b82da2b24d09ac47aceedd225013 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:12:11.147116 26796 raft_consensus.cc:729] T 4cade3193eac4883bce2f4599ffc7b68 P b440b82da2b24d09ac47aceedd225013 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b440b82da2b24d09ac47aceedd225013, State: Initialized, Role: FOLLOWER
I20260926 15:12:11.147710 26796 consensus_queue.cc:260] T 4cade3193eac4883bce2f4599ffc7b68 P b440b82da2b24d09ac47aceedd225013 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4067cd15627b4f5ba7f5eb07551e7fab" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 35363 } } peers { permanent_uuid: "cbe51bf32d864c85a4189dc5531617b1" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 37165 } } peers { permanent_uuid: "b440b82da2b24d09ac47aceedd225013" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 35357 } }
I20260926 15:12:11.149454 26776 heartbeater.cc:503] Master 127.12.34.254:43645 was elected leader, sending a full tablet report...
I20260926 15:12:11.149883 26796 ts_tablet_manager.cc:1428] T 4cade3193eac4883bce2f4599ffc7b68 P b440b82da2b24d09ac47aceedd225013: Time spent starting tablet: real 0.014s user 0.011s sys 0.004s
I20260926 15:12:11.204787 26800 raft_consensus.cc:482] T 4cade3193eac4883bce2f4599ffc7b68 P cbe51bf32d864c85a4189dc5531617b1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 15:12:11.205147 26800 raft_consensus.cc:504] T 4cade3193eac4883bce2f4599ffc7b68 P cbe51bf32d864c85a4189dc5531617b1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4067cd15627b4f5ba7f5eb07551e7fab" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 35363 } } peers { permanent_uuid: "cbe51bf32d864c85a4189dc5531617b1" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 37165 } } peers { permanent_uuid: "b440b82da2b24d09ac47aceedd225013" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 35357 } }
I20260926 15:12:11.206862 26800 leader_election.cc:288] T 4cade3193eac4883bce2f4599ffc7b68 P cbe51bf32d864c85a4189dc5531617b1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4067cd15627b4f5ba7f5eb07551e7fab (127.12.34.193:35363), b440b82da2b24d09ac47aceedd225013 (127.12.34.195:35357)
I20260926 15:12:11.211027 26801 raft_consensus.cc:482] T 4cade3193eac4883bce2f4599ffc7b68 P 4067cd15627b4f5ba7f5eb07551e7fab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 15:12:11.211295 26801 raft_consensus.cc:504] T 4cade3193eac4883bce2f4599ffc7b68 P 4067cd15627b4f5ba7f5eb07551e7fab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4067cd15627b4f5ba7f5eb07551e7fab" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 35363 } } peers { permanent_uuid: "cbe51bf32d864c85a4189dc5531617b1" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 37165 } } peers { permanent_uuid: "b440b82da2b24d09ac47aceedd225013" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 35357 } }
I20260926 15:12:11.213200 26801 leader_election.cc:288] T 4cade3193eac4883bce2f4599ffc7b68 P 4067cd15627b4f5ba7f5eb07551e7fab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cbe51bf32d864c85a4189dc5531617b1 (127.12.34.194:37165), b440b82da2b24d09ac47aceedd225013 (127.12.34.195:35357)
I20260926 15:12:11.215332 26793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:11.207347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.194:39849 (local address 127.12.34.195:35357)
0926 15:12:11.207579 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0926 15:12:11.207588 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 15:12:11.208681 (+ 1093us) server_negotiation.cc:407] Connection header received
0926 15:12:11.208834 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:12:11.208838 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:12:11.208914 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:12:11.209009 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:12:11.209680 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:12:11.210370 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:12:11.213020 (+ 2650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:12:11.214236 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:12:11.214418 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:12:11.214490 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 15:12:11.214841 (+ 351us) server_negotiation.cc:299] Negotiation successful
0926 15:12:11.215035 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260926 15:12:11.215315 26509 tablet.cc:2369] T 4cade3193eac4883bce2f4599ffc7b68 P 4067cd15627b4f5ba7f5eb07551e7fab: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 15:12:11.216636 26803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:11.207307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.195:35357 (local address 127.12.34.194:39849)
0926 15:12:11.208545 (+ 1238us) negotiation.cc:107] Waiting for socket to connect
0926 15:12:11.208564 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 15:12:11.208709 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:12:11.209041 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:12:11.209046 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:12:11.209107 (+ 61us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:12:11.209537 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:12:11.209549 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:12:11.210529 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:12:11.210533 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:12:11.212838 (+ 2305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:12:11.212853 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:12:11.214448 (+ 1595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:12:11.214454 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:12:11.214636 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:12:11.214673 (+ 37us) client_negotiation.cc:769] Sending connection context
0926 15:12:11.215892 (+ 1219us) client_negotiation.cc:240] Negotiation successful
0926 15:12:11.216079 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1109,"mutex_wait_us":52,"thread_start_us":870,"threads_started":1}
I20260926 15:12:11.217732 26731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cade3193eac4883bce2f4599ffc7b68" candidate_uuid: "cbe51bf32d864c85a4189dc5531617b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b440b82da2b24d09ac47aceedd225013" is_pre_election: true
I20260926 15:12:11.218051 26804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:11.207283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.193:35363 (local address 127.12.34.194:42405)
0926 15:12:11.208420 (+ 1137us) negotiation.cc:107] Waiting for socket to connect
0926 15:12:11.208439 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 15:12:11.208567 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:12:11.209000 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:12:11.209004 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:12:11.209028 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:12:11.209414 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:12:11.209423 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:12:11.210456 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:12:11.210462 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:12:11.212444 (+ 1982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:12:11.212460 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:12:11.217248 (+ 4788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:12:11.217254 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:12:11.217435 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:12:11.217470 (+ 35us) client_negotiation.cc:769] Sending connection context
0926 15:12:11.217568 (+ 98us) client_negotiation.cc:240] Negotiation successful
0926 15:12:11.217689 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1009,"mutex_wait_us":101,"thread_start_us":433,"threads_started":1}
I20260926 15:12:11.218382 26731 raft_consensus.cc:2454] T 4cade3193eac4883bce2f4599ffc7b68 P b440b82da2b24d09ac47aceedd225013 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cbe51bf32d864c85a4189dc5531617b1 in term 0.
I20260926 15:12:11.219378 26530 leader_election.cc:302] T 4cade3193eac4883bce2f4599ffc7b68 P cbe51bf32d864c85a4189dc5531617b1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b440b82da2b24d09ac47aceedd225013, cbe51bf32d864c85a4189dc5531617b1; no voters:
I20260926 15:12:11.219725 26790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:11.207279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.194:42405 (local address 127.12.34.193:35363)
0926 15:12:11.207494 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0926 15:12:11.207503 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 15:12:11.208535 (+ 1032us) server_negotiation.cc:407] Connection header received
0926 15:12:11.208748 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:12:11.208755 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:12:11.208846 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:12:11.208922 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:12:11.209560 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:12:11.210321 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:12:11.212605 (+ 2284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:12:11.213919 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:12:11.214068 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:12:11.214152 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 15:12:11.219233 (+ 5081us) server_negotiation.cc:299] Negotiation successful
0926 15:12:11.219464 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 15:12:11.219810 26800 raft_consensus.cc:2781] T 4cade3193eac4883bce2f4599ffc7b68 P cbe51bf32d864c85a4189dc5531617b1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 15:12:11.219964 26800 raft_consensus.cc:482] T 4cade3193eac4883bce2f4599ffc7b68 P cbe51bf32d864c85a4189dc5531617b1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 15:12:11.220079 26800 raft_consensus.cc:3037] T 4cade3193eac4883bce2f4599ffc7b68 P cbe51bf32d864c85a4189dc5531617b1 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:12:11.221706 26791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:11.213978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.193:33967 (local address 127.12.34.194:37165)
0926 15:12:11.214240 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0926 15:12:11.214244 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:12:11.214568 (+ 324us) server_negotiation.cc:407] Connection header received
0926 15:12:11.214770 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:12:11.214776 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:12:11.214854 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:12:11.214916 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:12:11.215794 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:12:11.216648 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:12:11.219216 (+ 2568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:12:11.220720 (+ 1504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:12:11.220861 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:12:11.220936 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 15:12:11.221299 (+ 363us) server_negotiation.cc:299] Negotiation successful
0926 15:12:11.221431 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146}
I20260926 15:12:11.221863 26805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:11.213397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.194:37165 (local address 127.12.34.193:33967)
0926 15:12:11.214476 (+ 1079us) negotiation.cc:107] Waiting for socket to connect
0926 15:12:11.214500 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 15:12:11.214620 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:12:11.215111 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:12:11.215115 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:12:11.215141 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:12:11.215626 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:12:11.215641 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:12:11.216849 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:12:11.216853 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:12:11.219005 (+ 2152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:12:11.219020 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:12:11.220914 (+ 1894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:12:11.220920 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:12:11.221134 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:12:11.221168 (+ 34us) client_negotiation.cc:769] Sending connection context
0926 15:12:11.221259 (+ 91us) client_negotiation.cc:240] Negotiation successful
0926 15:12:11.221389 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":960,"thread_start_us":1234,"threads_started":1}
I20260926 15:12:11.222544 26463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cade3193eac4883bce2f4599ffc7b68" candidate_uuid: "cbe51bf32d864c85a4189dc5531617b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4067cd15627b4f5ba7f5eb07551e7fab" is_pre_election: true
I20260926 15:12:11.222793 26597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cade3193eac4883bce2f4599ffc7b68" candidate_uuid: "4067cd15627b4f5ba7f5eb07551e7fab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cbe51bf32d864c85a4189dc5531617b1" is_pre_election: true
I20260926 15:12:11.222999 26463 raft_consensus.cc:2454] T 4cade3193eac4883bce2f4599ffc7b68 P 4067cd15627b4f5ba7f5eb07551e7fab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cbe51bf32d864c85a4189dc5531617b1 in term 0.
I20260926 15:12:11.223486 26800 raft_consensus.cc:504] T 4cade3193eac4883bce2f4599ffc7b68 P cbe51bf32d864c85a4189dc5531617b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4067cd15627b4f5ba7f5eb07551e7fab" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 35363 } } peers { permanent_uuid: "cbe51bf32d864c85a4189dc5531617b1" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 37165 } } peers { permanent_uuid: "b440b82da2b24d09ac47aceedd225013" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 35357 } }
I20260926 15:12:11.224037 26597 raft_consensus.cc:2379] T 4cade3193eac4883bce2f4599ffc7b68 P cbe51bf32d864c85a4189dc5531617b1 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4067cd15627b4f5ba7f5eb07551e7fab in current term 1: Already voted for candidate cbe51bf32d864c85a4189dc5531617b1 in this term.
I20260926 15:12:11.224642 26463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cade3193eac4883bce2f4599ffc7b68" candidate_uuid: "cbe51bf32d864c85a4189dc5531617b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4067cd15627b4f5ba7f5eb07551e7fab"
I20260926 15:12:11.224805 26800 leader_election.cc:288] T 4cade3193eac4883bce2f4599ffc7b68 P cbe51bf32d864c85a4189dc5531617b1 [CANDIDATE]: Term 1 election: Requested vote from peers 4067cd15627b4f5ba7f5eb07551e7fab (127.12.34.193:35363), b440b82da2b24d09ac47aceedd225013 (127.12.34.195:35357)
I20260926 15:12:11.224865 26463 raft_consensus.cc:3037] T 4cade3193eac4883bce2f4599ffc7b68 P 4067cd15627b4f5ba7f5eb07551e7fab [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:12:11.224962 26731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cade3193eac4883bce2f4599ffc7b68" candidate_uuid: "cbe51bf32d864c85a4189dc5531617b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b440b82da2b24d09ac47aceedd225013"
I20260926 15:12:11.225216 26731 raft_consensus.cc:3037] T 4cade3193eac4883bce2f4599ffc7b68 P b440b82da2b24d09ac47aceedd225013 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:12:11.226506 26793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:11.215366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.193:57957 (local address 127.12.34.195:35357)
0926 15:12:11.215845 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0926 15:12:11.215853 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 15:12:11.218999 (+ 3146us) server_negotiation.cc:407] Connection header received
0926 15:12:11.219233 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:12:11.219239 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:12:11.219303 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:12:11.219508 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:12:11.220296 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:12:11.221310 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:12:11.223806 (+ 2496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:12:11.225127 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:12:11.225305 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:12:11.225412 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 15:12:11.226073 (+ 661us) server_negotiation.cc:299] Negotiation successful
0926 15:12:11.226199 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":2904,"threads_started":1}
I20260926 15:12:11.226506 26806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:11.215032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.195:35357 (local address 127.12.34.193:57957)
0926 15:12:11.218901 (+ 3869us) negotiation.cc:107] Waiting for socket to connect
0926 15:12:11.218920 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 15:12:11.219091 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:12:11.219730 (+ 639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:12:11.219736 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:12:11.219767 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:12:11.220110 (+ 343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:12:11.220123 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:12:11.221478 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:12:11.221483 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:12:11.223614 (+ 2131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:12:11.223628 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:12:11.225727 (+ 2099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:12:11.225737 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:12:11.225909 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:12:11.225949 (+ 40us) client_negotiation.cc:769] Sending connection context
0926 15:12:11.226056 (+ 107us) client_negotiation.cc:240] Negotiation successful
0926 15:12:11.226178 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3752,"thread_start_us":3609,"threads_started":1}
I20260926 15:12:11.227363 26730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cade3193eac4883bce2f4599ffc7b68" candidate_uuid: "4067cd15627b4f5ba7f5eb07551e7fab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b440b82da2b24d09ac47aceedd225013" is_pre_election: true
I20260926 15:12:11.228118 26396 leader_election.cc:302] T 4cade3193eac4883bce2f4599ffc7b68 P 4067cd15627b4f5ba7f5eb07551e7fab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4067cd15627b4f5ba7f5eb07551e7fab; no voters: b440b82da2b24d09ac47aceedd225013, cbe51bf32d864c85a4189dc5531617b1
I20260926 15:12:11.228255 26463 raft_consensus.cc:2454] T 4cade3193eac4883bce2f4599ffc7b68 P 4067cd15627b4f5ba7f5eb07551e7fab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbe51bf32d864c85a4189dc5531617b1 in term 1.
I20260926 15:12:11.228493 26801 raft_consensus.cc:2726] T 4cade3193eac4883bce2f4599ffc7b68 P 4067cd15627b4f5ba7f5eb07551e7fab [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 15:12:11.228559 26731 raft_consensus.cc:2454] T 4cade3193eac4883bce2f4599ffc7b68 P b440b82da2b24d09ac47aceedd225013 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbe51bf32d864c85a4189dc5531617b1 in term 1.
I20260926 15:12:11.228901 26532 leader_election.cc:302] T 4cade3193eac4883bce2f4599ffc7b68 P cbe51bf32d864c85a4189dc5531617b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4067cd15627b4f5ba7f5eb07551e7fab, cbe51bf32d864c85a4189dc5531617b1; no voters:
I20260926 15:12:11.229230 26800 raft_consensus.cc:2781] T 4cade3193eac4883bce2f4599ffc7b68 P cbe51bf32d864c85a4189dc5531617b1 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:12:11.229614 26800 raft_consensus.cc:686] T 4cade3193eac4883bce2f4599ffc7b68 P cbe51bf32d864c85a4189dc5531617b1 [term 1 LEADER]: Becoming Leader. State: Replica: cbe51bf32d864c85a4189dc5531617b1, State: Running, Role: LEADER
I20260926 15:12:11.230041 26800 consensus_queue.cc:237] T 4cade3193eac4883bce2f4599ffc7b68 P cbe51bf32d864c85a4189dc5531617b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4067cd15627b4f5ba7f5eb07551e7fab" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 35363 } } peers { permanent_uuid: "cbe51bf32d864c85a4189dc5531617b1" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 37165 } } peers { permanent_uuid: "b440b82da2b24d09ac47aceedd225013" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 35357 } }
I20260926 15:12:11.234529 26318 catalog_manager.cc:5495] T 4cade3193eac4883bce2f4599ffc7b68 P cbe51bf32d864c85a4189dc5531617b1 reported cstate change: term changed from 0 to 1, leader changed from <none> to cbe51bf32d864c85a4189dc5531617b1 (127.12.34.194). New cstate: current_term: 1 leader_uuid: "cbe51bf32d864c85a4189dc5531617b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4067cd15627b4f5ba7f5eb07551e7fab" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 35363 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cbe51bf32d864c85a4189dc5531617b1" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 37165 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b440b82da2b24d09ac47aceedd225013" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 35357 } health_report { overall_health: UNKNOWN } } }
W20260926 15:12:11.250638 26643 tablet.cc:2369] T 4cade3193eac4883bce2f4599ffc7b68 P cbe51bf32d864c85a4189dc5531617b1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 15:12:11.256379 26791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:11.251187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57350 (local address 127.12.34.194:37165)
0926 15:12:11.251366 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0926 15:12:11.251373 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:12:11.251471 (+ 98us) server_negotiation.cc:407] Connection header received
0926 15:12:11.251627 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:12:11.251631 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:12:11.251700 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:12:11.251819 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 15:12:11.252147 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:12:11.252855 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:12:11.254229 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:12:11.254625 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:12:11.254774 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:12:11.255296 (+ 522us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 15:12:11.255756 (+ 460us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 15:12:11.256007 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0926 15:12:11.256104 (+ 97us) server_negotiation.cc:299] Negotiation successful
0926 15:12:11.256169 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 15:12:11.263818 26731 raft_consensus.cc:1261] T 4cade3193eac4883bce2f4599ffc7b68 P b440b82da2b24d09ac47aceedd225013 [term 1 FOLLOWER]: Refusing update from remote peer cbe51bf32d864c85a4189dc5531617b1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:12:11.263922 26463 raft_consensus.cc:1261] T 4cade3193eac4883bce2f4599ffc7b68 P 4067cd15627b4f5ba7f5eb07551e7fab [term 1 FOLLOWER]: Refusing update from remote peer cbe51bf32d864c85a4189dc5531617b1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:12:11.264635 26808 consensus_queue.cc:1035] T 4cade3193eac4883bce2f4599ffc7b68 P cbe51bf32d864c85a4189dc5531617b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b440b82da2b24d09ac47aceedd225013" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 35357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:12:11.264945 26800 consensus_queue.cc:1035] T 4cade3193eac4883bce2f4599ffc7b68 P cbe51bf32d864c85a4189dc5531617b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4067cd15627b4f5ba7f5eb07551e7fab" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 35363 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 15:12:11.271588 26777 tablet.cc:2369] T 4cade3193eac4883bce2f4599ffc7b68 P b440b82da2b24d09ac47aceedd225013: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 15:12:11.275602 26811 mvcc.cc:205] Tried to move back new op lower bound from 7333623936048046080 to 7333623935921999872. Current Snapshot: MvccSnapshot[applied={T|T < 7333623936048046080}]
I20260926 15:12:11.288684 26319 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44654:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 15:12:11.288964 26319 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44654:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 15:12:11.292554 26319 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2e76b19894564aa28e81f161acd96f87: Sending DeleteTablet for 3 replicas of tablet 4cade3193eac4883bce2f4599ffc7b68
I20260926 15:12:11.293624 26443 tablet_service.cc:1514] Processing DeleteTablet for tablet 4cade3193eac4883bce2f4599ffc7b68 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:12:11 UTC) from {username='kudu', principal='kudu/127.12.34.254@KRBTEST.COM'} at 127.0.0.1:43732
I20260926 15:12:11.293830 26577 tablet_service.cc:1514] Processing DeleteTablet for tablet 4cade3193eac4883bce2f4599ffc7b68 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:12:11 UTC) from {username='kudu', principal='kudu/127.12.34.254@KRBTEST.COM'} at 127.0.0.1:57348
I20260926 15:12:11.293874 26711 tablet_service.cc:1514] Processing DeleteTablet for tablet 4cade3193eac4883bce2f4599ffc7b68 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:12:11 UTC) from {username='kudu', principal='kudu/127.12.34.254@KRBTEST.COM'} at 127.0.0.1:57602
I20260926 15:12:11.295003 12427 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 26380
I20260926 15:12:11.308712 26821 tablet_replica.cc:330] T 4cade3193eac4883bce2f4599ffc7b68 P 4067cd15627b4f5ba7f5eb07551e7fab: stopping tablet replica
I20260926 15:12:11.309437 26821 raft_consensus.cc:2229] T 4cade3193eac4883bce2f4599ffc7b68 P 4067cd15627b4f5ba7f5eb07551e7fab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 15:12:11.310099 26823 tablet_replica.cc:330] T 4cade3193eac4883bce2f4599ffc7b68 P b440b82da2b24d09ac47aceedd225013: stopping tablet replica
I20260926 15:12:11.310096 26821 raft_consensus.cc:2258] T 4cade3193eac4883bce2f4599ffc7b68 P 4067cd15627b4f5ba7f5eb07551e7fab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:12:11.310623 26823 raft_consensus.cc:2229] T 4cade3193eac4883bce2f4599ffc7b68 P b440b82da2b24d09ac47aceedd225013 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 15:12:11.311144 26823 raft_consensus.cc:2258] T 4cade3193eac4883bce2f4599ffc7b68 P b440b82da2b24d09ac47aceedd225013 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:12:11.313181 26823 ts_tablet_manager.cc:1905] T 4cade3193eac4883bce2f4599ffc7b68 P b440b82da2b24d09ac47aceedd225013: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:12:11.313947 26822 tablet_replica.cc:330] T 4cade3193eac4883bce2f4599ffc7b68 P cbe51bf32d864c85a4189dc5531617b1: stopping tablet replica
I20260926 15:12:11.314419 26822 raft_consensus.cc:2229] T 4cade3193eac4883bce2f4599ffc7b68 P cbe51bf32d864c85a4189dc5531617b1 [term 1 LEADER]: Raft consensus shutting down.
I20260926 15:12:11.315073 26822 raft_consensus.cc:2258] T 4cade3193eac4883bce2f4599ffc7b68 P cbe51bf32d864c85a4189dc5531617b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:12:11.317446 26821 ts_tablet_manager.cc:1905] T 4cade3193eac4883bce2f4599ffc7b68 P 4067cd15627b4f5ba7f5eb07551e7fab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:12:11.317533 26822 ts_tablet_manager.cc:1905] T 4cade3193eac4883bce2f4599ffc7b68 P cbe51bf32d864c85a4189dc5531617b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:12:11.320113 26823 ts_tablet_manager.cc:1918] T 4cade3193eac4883bce2f4599ffc7b68 P b440b82da2b24d09ac47aceedd225013: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 15:12:11.320329 26823 log.cc:1196] T 4cade3193eac4883bce2f4599ffc7b68 P b440b82da2b24d09ac47aceedd225013: Deleting WAL directory at /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/ts-2/wal/wals/4cade3193eac4883bce2f4599ffc7b68
I20260926 15:12:11.320921 26823 ts_tablet_manager.cc:1939] T 4cade3193eac4883bce2f4599ffc7b68 P b440b82da2b24d09ac47aceedd225013: Deleting consensus metadata
I20260926 15:12:11.322813 26303 catalog_manager.cc:4841] TS b440b82da2b24d09ac47aceedd225013 (127.12.34.195:35357): tablet 4cade3193eac4883bce2f4599ffc7b68 (table test-table [id=201d197185dd48eda45cabcc6ae97d1b]) successfully deleted
I20260926 15:12:11.325273 26821 ts_tablet_manager.cc:1918] T 4cade3193eac4883bce2f4599ffc7b68 P 4067cd15627b4f5ba7f5eb07551e7fab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 15:12:11.325503 26821 log.cc:1196] T 4cade3193eac4883bce2f4599ffc7b68 P 4067cd15627b4f5ba7f5eb07551e7fab: Deleting WAL directory at /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/ts-0/wal/wals/4cade3193eac4883bce2f4599ffc7b68
Sep 26 15:12:11 dist-test-slave-865v krb5kdc[26274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435528, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.34.193@KRBTEST.COM
I20260926 15:12:11.326097 26821 ts_tablet_manager.cc:1939] T 4cade3193eac4883bce2f4599ffc7b68 P 4067cd15627b4f5ba7f5eb07551e7fab: Deleting consensus metadata
I20260926 15:12:11.326869 26822 ts_tablet_manager.cc:1918] T 4cade3193eac4883bce2f4599ffc7b68 P cbe51bf32d864c85a4189dc5531617b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 15:12:11.327100 26822 log.cc:1196] T 4cade3193eac4883bce2f4599ffc7b68 P cbe51bf32d864c85a4189dc5531617b1: Deleting WAL directory at /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/ts-1/wal/wals/4cade3193eac4883bce2f4599ffc7b68
I20260926 15:12:11.327648 26822 ts_tablet_manager.cc:1939] T 4cade3193eac4883bce2f4599ffc7b68 P cbe51bf32d864c85a4189dc5531617b1: Deleting consensus metadata
I20260926 15:12:11.328832 26305 catalog_manager.cc:4841] TS 4067cd15627b4f5ba7f5eb07551e7fab (127.12.34.193:35363): tablet 4cade3193eac4883bce2f4599ffc7b68 (table test-table [id=201d197185dd48eda45cabcc6ae97d1b]) successfully deleted
I20260926 15:12:11.329209 26303 catalog_manager.cc:4841] TS cbe51bf32d864c85a4189dc5531617b1 (127.12.34.194:37165): tablet 4cade3193eac4883bce2f4599ffc7b68 (table test-table [id=201d197185dd48eda45cabcc6ae97d1b]) successfully deleted
I20260926 15:12:11.334169 26790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:11.295828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43746 (local address 127.12.34.193:35363)
0926 15:12:11.296010 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0926 15:12:11.296019 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 15:12:11.303488 (+ 7469us) server_negotiation.cc:407] Connection header received
0926 15:12:11.304539 (+ 1051us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:12:11.304560 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:12:11.304695 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:12:11.304787 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:12:11.310483 (+ 5696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:12:11.311666 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:12:11.313880 (+ 2214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:12:11.314498 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:12:11.314633 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:12:11.326809 (+ 12176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:12:11.326848 (+ 39us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:12:11.326867 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:12:11.326918 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:12:11.330984 (+ 4066us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:12:11.331798 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:12:11.331807 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:12:11.331811 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:12:11.331971 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:12:11.332383 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:12:11.332389 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:12:11.332392 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:12:11.332831 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:12:11.333176 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0926 15:12:11.333554 (+ 378us) server_negotiation.cc:299] Negotiation successful
0926 15:12:11.333785 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 15:12:11.334815 26503 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:12:11.449283 12427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8uoJGv/build/asan/bin/kudu with pid 26380
I20260926 15:12:11.473342 12427 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 26514
Sep 26 15:12:11 dist-test-slave-865v krb5kdc[26274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435528, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.34.194@KRBTEST.COM
I20260926 15:12:11.490548 26791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:11.474219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57360 (local address 127.12.34.194:37165)
0926 15:12:11.474481 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0926 15:12:11.474489 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 15:12:11.474518 (+ 29us) server_negotiation.cc:407] Connection header received
0926 15:12:11.474573 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:12:11.474576 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:12:11.474640 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:12:11.474761 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:12:11.476029 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:12:11.476774 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:12:11.478119 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:12:11.478549 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:12:11.478690 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:12:11.483202 (+ 4512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:12:11.483215 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:12:11.483231 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:12:11.483272 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:12:11.487130 (+ 3858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:12:11.488223 (+ 1093us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:12:11.488231 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:12:11.488234 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:12:11.488403 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:12:11.488819 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:12:11.488825 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:12:11.488828 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:12:11.489328 (+ 500us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:12:11.489687 (+ 359us) server_negotiation.cc:1035] Waiting for connection context
0926 15:12:11.489953 (+ 266us) server_negotiation.cc:299] Negotiation successful
0926 15:12:11.490216 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 15:12:11.491344 26637 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:12:11.607189 12427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8uoJGv/build/asan/bin/kudu with pid 26514
I20260926 15:12:11.626062 12427 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 26648
Sep 26 15:12:11 dist-test-slave-865v krb5kdc[26274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435528, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.34.195@KRBTEST.COM
I20260926 15:12:11.640729 26793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:11.626817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57610 (local address 127.12.34.195:35357)
0926 15:12:11.626992 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0926 15:12:11.626996 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:12:11.627031 (+ 35us) server_negotiation.cc:407] Connection header received
0926 15:12:11.627175 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:12:11.627178 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:12:11.627247 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:12:11.627327 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:12:11.628616 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:12:11.629337 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:12:11.630547 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:12:11.630941 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:12:11.631038 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:12:11.635241 (+ 4203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:12:11.635253 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:12:11.635268 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:12:11.635298 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:12:11.638156 (+ 2858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:12:11.638797 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:12:11.638805 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:12:11.638807 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:12:11.638945 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:12:11.639296 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:12:11.639299 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:12:11.639301 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:12:11.639655 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:12:11.639856 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0926 15:12:11.640220 (+ 364us) server_negotiation.cc:299] Negotiation successful
0926 15:12:11.640374 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 15:12:11.641496 26771 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:12:11.733649 12427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8uoJGv/build/asan/bin/kudu with pid 26648
I20260926 15:12:11.758484 12427 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 26287
I20260926 15:12:11.759652 26349 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:12:11.874399 12427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8uoJGv/build/asan/bin/kudu with pid 26287
I20260926 15:12:11.895453 12427 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8uoJGv/build/asan/bin/kudu
/tmp/dist-test-task8uoJGv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.34.254:43645
--webserver_interface=127.12.34.254
--webserver_port=42385
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.254.keytab
--principal=kudu/127.12.34.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.34.212:36007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.34.254:43645
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:12:12.214234 26829 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:12:12.214607 26829 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:12:12.214769 26829 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:12:12.221909 26829 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:12:12.222046 26829 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:12:12.222151 26829 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:12:12.222286 26829 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:12:12.222338 26829 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:12:12.232048 26829 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.34.212:36007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.12.34.254:43645
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.254.keytab
--principal=kudu/127.12.34.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.34.254:43645
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.34.254
--webserver_port=42385
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:12:12.233332 26829 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:12:12.234622 26829 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:12:12.257375 26838 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:12:12.257556 26835 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:12:12.257499 26836 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:12:12.258435 26829 server_base.cc:1034] running on GCE node
I20260926 15:12:12.259253 26829 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:12:12.260741 26829 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:12:12.261961 26829 hybrid_clock.cc:648] HybridClock initialized: now 1790435532261918 us; error 32 us; skew 500 ppm
I20260926 15:12:12.262393 26829 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:12:12 dist-test-slave-865v krb5kdc[26274](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435532, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:12:12.266014 26829 init.cc:377] Logged in from keytab as kudu/127.12.34.254@KRBTEST.COM (short username kudu)
I20260926 15:12:12.267587 26829 webserver.cc:460] Webserver started at http://127.12.34.254:42385/ using document root <none> and password file <none>
I20260926 15:12:12.268311 26829 fs_manager.cc:359] Metadata directory not provided
I20260926 15:12:12.268451 26829 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:12:12.274134 26829 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 15:12:12.276960 26845 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:12:12.278491 26829 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 15:12:12.278704 26829 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/master-0/data,/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/master-0/wal
uuid: "2e76b19894564aa28e81f161acd96f87"
format_stamp: "Formatted at 2026-09-26 15:12:09 on dist-test-slave-865v"
server_key: "4cd98550f90faf5dce91d40b3617c5c3"
server_key_iv: "5bc885476fa2d48422d0345de0ed7477"
server_key_version: "encryptionkey@0"
I20260926 15:12:12.279891 26829 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:12:12.300909 26829 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:12:12.305217 26829 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:12:12.305501 26829 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:12:12.324818 26829 rpc_server.cc:307] RPC server started. Bound to: 127.12.34.254:43645
I20260926 15:12:12.324853 26896 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.34.254:43645 every 8 connection(s)
I20260926 15:12:12.326345 26829 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
I20260926 15:12:12.333226 26897 sys_catalog.cc:263] Verifying existing consensus state
I20260926 15:12:12.336484 26897 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e76b19894564aa28e81f161acd96f87: Bootstrap starting.
I20260926 15:12:12.336867 12427 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8uoJGv/build/asan/bin/kudu as pid 26829
I20260926 15:12:12.337826 12427 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8uoJGv/build/asan/bin/kudu
/tmp/dist-test-task8uoJGv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.34.193:35363
--local_ip_for_outbound_sockets=127.12.34.193
--tserver_master_addrs=127.12.34.254:43645
--webserver_port=33943
--webserver_interface=127.12.34.193
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.193.keytab
--principal=kudu/127.12.34.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.34.212:36007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 15:12:12.359226 26897 log.cc:824] T 00000000000000000000000000000000 P 2e76b19894564aa28e81f161acd96f87: Log is configured to *not* fsync() on all Append() calls
I20260926 15:12:12.371372 26897 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e76b19894564aa28e81f161acd96f87: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 15:12:12.371945 26897 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e76b19894564aa28e81f161acd96f87: Bootstrap complete.
I20260926 15:12:12.381330 26897 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2e76b19894564aa28e81f161acd96f87 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e76b19894564aa28e81f161acd96f87" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 43645 } }
I20260926 15:12:12.382162 26897 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2e76b19894564aa28e81f161acd96f87 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e76b19894564aa28e81f161acd96f87, State: Initialized, Role: FOLLOWER
I20260926 15:12:12.382741 26897 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2e76b19894564aa28e81f161acd96f87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e76b19894564aa28e81f161acd96f87" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 43645 } }
I20260926 15:12:12.382983 26897 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2e76b19894564aa28e81f161acd96f87 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 15:12:12.383118 26897 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2e76b19894564aa28e81f161acd96f87 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 15:12:12.383296 26897 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2e76b19894564aa28e81f161acd96f87 [term 1 FOLLOWER]: Advancing to term 2
I20260926 15:12:12.387259 26897 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2e76b19894564aa28e81f161acd96f87 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e76b19894564aa28e81f161acd96f87" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 43645 } }
I20260926 15:12:12.387759 26897 leader_election.cc:302] T 00000000000000000000000000000000 P 2e76b19894564aa28e81f161acd96f87 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2e76b19894564aa28e81f161acd96f87; no voters:
I20260926 15:12:12.388401 26897 leader_election.cc:288] T 00000000000000000000000000000000 P 2e76b19894564aa28e81f161acd96f87 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 15:12:12.388553 26901 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2e76b19894564aa28e81f161acd96f87 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 15:12:12.390913 26897 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2e76b19894564aa28e81f161acd96f87 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:12:12.395532 26901 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2e76b19894564aa28e81f161acd96f87 [term 2 LEADER]: Becoming Leader. State: Replica: 2e76b19894564aa28e81f161acd96f87, State: Running, Role: LEADER
I20260926 15:12:12.396378 26901 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2e76b19894564aa28e81f161acd96f87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e76b19894564aa28e81f161acd96f87" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 43645 } }
I20260926 15:12:12.399276 26902 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e76b19894564aa28e81f161acd96f87 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "2e76b19894564aa28e81f161acd96f87" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e76b19894564aa28e81f161acd96f87" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 43645 } } }
I20260926 15:12:12.399711 26902 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e76b19894564aa28e81f161acd96f87 [sys.catalog]: This master's current role is: LEADER
I20260926 15:12:12.400321 26903 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e76b19894564aa28e81f161acd96f87 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2e76b19894564aa28e81f161acd96f87. Latest consensus state: current_term: 2 leader_uuid: "2e76b19894564aa28e81f161acd96f87" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e76b19894564aa28e81f161acd96f87" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 43645 } } }
I20260926 15:12:12.400647 26903 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e76b19894564aa28e81f161acd96f87 [sys.catalog]: This master's current role is: LEADER
I20260926 15:12:12.410158 26918 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 15:12:12.410349 26916 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:12:12.416219 26916 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 15:12:12.417332 26916 catalog_manager.cc:1259] Loaded cluster ID: 5b907a4a881e41c48bab7c4645ef1315
I20260926 15:12:12.417433 26916 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 15:12:12.419582 26916 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260926 15:12:12.430265 26916 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260926 15:12:12.437054 26916 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:12:12.438338 26916 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2e76b19894564aa28e81f161acd96f87: Loaded TSK: 0
I20260926 15:12:12.439399 26916 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 15:12:12.704550 26899 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:12:12.704799 26899 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:12:12.704950 26899 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:12:12.712204 26899 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:12:12.712299 26899 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:12:12.712505 26899 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.34.193
I20260926 15:12:12.724802 26899 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.34.212:36007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.193.keytab
--principal=kudu/127.12.34.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.34.193:35363
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.34.193
--webserver_port=33943
--tserver_master_addrs=127.12.34.254:43645
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.34.193
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:12:12.726715 26899 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:12:12.728775 26899 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:12:12.744251 26928 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:12:12.744230 26926 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:12:12.744607 26925 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:12:12.745690 26899 server_base.cc:1034] running on GCE node
I20260926 15:12:12.746464 26899 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:12:12.747700 26899 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:12:12.748962 26899 hybrid_clock.cc:648] HybridClock initialized: now 1790435532748916 us; error 61 us; skew 500 ppm
I20260926 15:12:12.749372 26899 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:12:12 dist-test-slave-865v krb5kdc[26274](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435532, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:12:12.761584 26899 init.cc:377] Logged in from keytab as kudu/127.12.34.193@KRBTEST.COM (short username kudu)
I20260926 15:12:12.763470 26899 webserver.cc:460] Webserver started at http://127.12.34.193:33943/ using document root <none> and password file <none>
I20260926 15:12:12.764392 26899 fs_manager.cc:359] Metadata directory not provided
I20260926 15:12:12.764554 26899 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:12:12.771859 26899 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 15:12:12.775513 26935 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:12:12.777215 26899 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 15:12:12.777385 26899 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/ts-0/data,/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/ts-0/wal
uuid: "4067cd15627b4f5ba7f5eb07551e7fab"
format_stamp: "Formatted at 2026-09-26 15:12:09 on dist-test-slave-865v"
server_key: "26f9593fa47ac2148c4a20256e4c7f86"
server_key_iv: "2c7606219edfd0d3e9eea39aa26ad591"
server_key_version: "encryptionkey@0"
I20260926 15:12:12.778752 26899 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:12:12.819538 26899 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:12:12.824039 26899 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:12:12.824388 26899 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:12:12.825392 26899 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:12:12.826689 26899 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:12:12.826810 26899 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:12:12.826946 26899 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:12:12.827041 26899 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:12:12.860050 26899 rpc_server.cc:307] RPC server started. Bound to: 127.12.34.193:35363
I20260926 15:12:12.860131 27047 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.34.193:35363 every 8 connection(s)
I20260926 15:12:12.861603 26899 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/ts-0/data/info.pb
I20260926 15:12:12.863715 12427 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8uoJGv/build/asan/bin/kudu as pid 26899
I20260926 15:12:12.864461 12427 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8uoJGv/build/asan/bin/kudu
/tmp/dist-test-task8uoJGv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.34.194:37165
--local_ip_for_outbound_sockets=127.12.34.194
--tserver_master_addrs=127.12.34.254:43645
--webserver_port=38851
--webserver_interface=127.12.34.194
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.194.keytab
--principal=kudu/127.12.34.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.34.212:36007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 15:12:12 dist-test-slave-865v krb5kdc[26274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435532, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.193@KRBTEST.COM for kudu/127.12.34.254@KRBTEST.COM
I20260926 15:12:12.888584 27050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:12.864430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.193:54947 (local address 127.12.34.254:43645)
0926 15:12:12.869560 (+ 5130us) server_negotiation.cc:206] Beginning negotiation
0926 15:12:12.869576 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 15:12:12.869623 (+ 47us) server_negotiation.cc:407] Connection header received
0926 15:12:12.869972 (+ 349us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:12:12.869991 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:12:12.870399 (+ 408us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:12:12.870789 (+ 390us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:12:12.872675 (+ 1886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:12:12.873921 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:12:12.876350 (+ 2429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:12:12.876929 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:12:12.877052 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:12:12.881065 (+ 4013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:12:12.881115 (+ 50us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:12:12.881136 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:12:12.881179 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:12:12.884284 (+ 3105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:12:12.885040 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:12:12.885048 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:12:12.885054 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:12:12.885250 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:12:12.885672 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:12:12.885679 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:12:12.885682 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:12:12.886101 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:12:12.886297 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0926 15:12:12.887031 (+ 734us) server_negotiation.cc:299] Negotiation successful
0926 15:12:12.887373 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4892,"thread_start_us":4781,"threads_started":1}
I20260926 15:12:12.888574 27051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:12.864770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:43645 (local address 127.12.34.193:54947)
0926 15:12:12.865816 (+ 1046us) negotiation.cc:107] Waiting for socket to connect
0926 15:12:12.865872 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0926 15:12:12.869068 (+ 3196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:12:12.870927 (+ 1859us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:12:12.870940 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:12:12.871674 (+ 734us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:12:12.872461 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:12:12.872487 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:12:12.874071 (+ 1584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:12:12.874079 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:12:12.876210 (+ 2131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:12:12.876228 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:12:12.877173 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:12:12.877183 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:12:12.877501 (+ 318us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:12:12.878224 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:12:12.878256 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:12:12.880804 (+ 2548us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:12:12.884418 (+ 3614us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:12:12.884426 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:12:12.884438 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:12:12.884874 (+ 436us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:12:12.885370 (+ 496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:12:12.885375 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:12:12.885379 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:12:12.885560 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:12:12.886258 (+ 698us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:12:12.886265 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:12:12.886564 (+ 299us) client_negotiation.cc:769] Sending connection context
0926 15:12:12.886816 (+ 252us) client_negotiation.cc:240] Negotiation successful
0926 15:12:12.887349 (+ 533us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":673,"thread_start_us":367,"threads_started":1}
I20260926 15:12:12.891499 27048 heartbeater.cc:347] Connected to a master server at 127.12.34.254:43645
I20260926 15:12:12.891825 27048 heartbeater.cc:464] Registering TS with master...
I20260926 15:12:12.892575 27048 heartbeater.cc:511] Master 127.12.34.254:43645 requested a full tablet report, sending...
I20260926 15:12:12.894923 26862 ts_manager.cc:194] Registered new tserver with Master: 4067cd15627b4f5ba7f5eb07551e7fab (127.12.34.193:35363)
I20260926 15:12:12.896952 26862 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.34.193@KRBTEST.COM'} at 127.12.34.193:54947
W20260926 15:12:13.188030 27052 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:12:13.188306 27052 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:12:13.188413 27052 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:12:13.196826 27052 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:12:13.196942 27052 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:12:13.197063 27052 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.34.194
I20260926 15:12:13.207566 27052 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.34.212:36007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.194.keytab
--principal=kudu/127.12.34.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.34.194:37165
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.34.194
--webserver_port=38851
--tserver_master_addrs=127.12.34.254:43645
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.34.194
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:12:13.208882 27052 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:12:13.210261 27052 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:12:13.223278 27057 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:12:13.223387 27060 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:12:13.223332 27058 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:12:13.224385 27052 server_base.cc:1034] running on GCE node
I20260926 15:12:13.225028 27052 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:12:13.226109 27052 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:12:13.227367 27052 hybrid_clock.cc:648] HybridClock initialized: now 1790435533227306 us; error 34 us; skew 500 ppm
I20260926 15:12:13.227660 27052 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:12:13 dist-test-slave-865v krb5kdc[26274](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435533, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:12:13.231109 27052 init.cc:377] Logged in from keytab as kudu/127.12.34.194@KRBTEST.COM (short username kudu)
I20260926 15:12:13.232586 27052 webserver.cc:460] Webserver started at http://127.12.34.194:38851/ using document root <none> and password file <none>
I20260926 15:12:13.233314 27052 fs_manager.cc:359] Metadata directory not provided
I20260926 15:12:13.233469 27052 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:12:13.238698 27052 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.003s
I20260926 15:12:13.241515 27067 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:12:13.242775 27052 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 15:12:13.242944 27052 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/ts-1/data,/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/ts-1/wal
uuid: "cbe51bf32d864c85a4189dc5531617b1"
format_stamp: "Formatted at 2026-09-26 15:12:10 on dist-test-slave-865v"
server_key: "b57409c04ecb3bf07d4f7f77e12c6e23"
server_key_iv: "807e84f89a714d99727fafcd477a3854"
server_key_version: "encryptionkey@0"
I20260926 15:12:13.243925 27052 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:12:13.268038 27052 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:12:13.272012 27052 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:12:13.272289 27052 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:12:13.273330 27052 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:12:13.274647 27052 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:12:13.274766 27052 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:12:13.274906 27052 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:12:13.274998 27052 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:12:13.310286 27052 rpc_server.cc:307] RPC server started. Bound to: 127.12.34.194:37165
I20260926 15:12:13.310381 27179 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.34.194:37165 every 8 connection(s)
I20260926 15:12:13.311715 27052 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/ts-1/data/info.pb
I20260926 15:12:13.318847 12427 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8uoJGv/build/asan/bin/kudu as pid 27052
I20260926 15:12:13.319517 12427 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8uoJGv/build/asan/bin/kudu
/tmp/dist-test-task8uoJGv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.34.195:35357
--local_ip_for_outbound_sockets=127.12.34.195
--tserver_master_addrs=127.12.34.254:43645
--webserver_port=35437
--webserver_interface=127.12.34.195
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.195.keytab
--principal=kudu/127.12.34.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.34.212:36007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 15:12:13 dist-test-slave-865v krb5kdc[26274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435533, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.194@KRBTEST.COM for kudu/127.12.34.254@KRBTEST.COM
I20260926 15:12:13.332973 27050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:13.314279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.194:50767 (local address 127.12.34.254:43645)
0926 15:12:13.314454 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0926 15:12:13.314458 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:12:13.315617 (+ 1159us) server_negotiation.cc:407] Connection header received
0926 15:12:13.317701 (+ 2084us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:12:13.317707 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:12:13.317800 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:12:13.317918 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:12:13.319531 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:12:13.320319 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:12:13.321922 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:12:13.322334 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:12:13.322465 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:12:13.326491 (+ 4026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:12:13.326497 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:12:13.326500 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:12:13.326535 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:12:13.329398 (+ 2863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:12:13.330114 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:12:13.330126 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:12:13.330129 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:12:13.330247 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:12:13.330680 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:12:13.330685 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:12:13.330688 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:12:13.330962 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:12:13.331333 (+ 371us) server_negotiation.cc:1035] Waiting for connection context
0926 15:12:13.332326 (+ 993us) server_negotiation.cc:299] Negotiation successful
0926 15:12:13.332600 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 15:12:13.334129 27182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:13.314735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:43645 (local address 127.12.34.194:50767)
0926 15:12:13.315451 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0926 15:12:13.315489 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 15:12:13.317421 (+ 1932us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:12:13.318149 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:12:13.318158 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:12:13.318659 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:12:13.319265 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:12:13.319283 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:12:13.320492 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:12:13.320496 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:12:13.321767 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:12:13.321779 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:12:13.322586 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:12:13.322594 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:12:13.322814 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:12:13.323516 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:12:13.323550 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:12:13.326189 (+ 2639us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:12:13.329541 (+ 3352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:12:13.329550 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:12:13.329564 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:12:13.329980 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:12:13.330355 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:12:13.330360 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:12:13.330364 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:12:13.330564 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:12:13.331294 (+ 730us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:12:13.331301 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:12:13.331845 (+ 544us) client_negotiation.cc:769] Sending connection context
0926 15:12:13.332135 (+ 290us) client_negotiation.cc:240] Negotiation successful
0926 15:12:13.332769 (+ 634us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":300,"threads_started":1}
I20260926 15:12:13.335577 27180 heartbeater.cc:347] Connected to a master server at 127.12.34.254:43645
I20260926 15:12:13.335899 27180 heartbeater.cc:464] Registering TS with master...
I20260926 15:12:13.336615 27180 heartbeater.cc:511] Master 127.12.34.254:43645 requested a full tablet report, sending...
I20260926 15:12:13.338140 26862 ts_manager.cc:194] Registered new tserver with Master: cbe51bf32d864c85a4189dc5531617b1 (127.12.34.194:37165)
I20260926 15:12:13.338972 26862 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.34.194@KRBTEST.COM'} at 127.12.34.194:50767
W20260926 15:12:13.646100 27183 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:12:13.646369 27183 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:12:13.646467 27183 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:12:13.653636 27183 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:12:13.653733 27183 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:12:13.653841 27183 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.34.195
I20260926 15:12:13.663789 27183 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.34.212:36007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.195.keytab
--principal=kudu/127.12.34.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.34.195:35357
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.34.195
--webserver_port=35437
--tserver_master_addrs=127.12.34.254:43645
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.34.195
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:12:13.664990 27183 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:12:13.666282 27183 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:12:13.679111 27189 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:12:13.679083 27191 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:12:13.679282 27188 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:12:13.679759 27183 server_base.cc:1034] running on GCE node
I20260926 15:12:13.680356 27183 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:12:13.681320 27183 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:12:13.682546 27183 hybrid_clock.cc:648] HybridClock initialized: now 1790435533682498 us; error 29 us; skew 500 ppm
I20260926 15:12:13.682888 27183 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:12:13 dist-test-slave-865v krb5kdc[26274](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435533, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:12:13.686321 27183 init.cc:377] Logged in from keytab as kudu/127.12.34.195@KRBTEST.COM (short username kudu)
I20260926 15:12:13.687846 27183 webserver.cc:460] Webserver started at http://127.12.34.195:35437/ using document root <none> and password file <none>
I20260926 15:12:13.688544 27183 fs_manager.cc:359] Metadata directory not provided
I20260926 15:12:13.688704 27183 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:12:13.694787 27183 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 15:12:13.697597 27198 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:12:13.698870 27183 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 15:12:13.699081 27183 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/ts-2/data,/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/ts-2/wal
uuid: "b440b82da2b24d09ac47aceedd225013"
format_stamp: "Formatted at 2026-09-26 15:12:10 on dist-test-slave-865v"
server_key: "12aec6bcbdef59bc94e5bbd9e3b0a73b"
server_key_iv: "5a9e03d0524afb3807e1879b21bc934c"
server_key_version: "encryptionkey@0"
I20260926 15:12:13.700021 27183 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:12:13.716871 27183 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:12:13.721022 27183 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:12:13.721310 27183 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:12:13.722262 27183 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:12:13.723559 27183 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:12:13.723681 27183 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:12:13.723819 27183 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:12:13.723907 27183 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:12:13.759810 27183 rpc_server.cc:307] RPC server started. Bound to: 127.12.34.195:35357
I20260926 15:12:13.759894 27310 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.34.195:35357 every 8 connection(s)
I20260926 15:12:13.761312 27183 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/ts-2/data/info.pb
I20260926 15:12:13.768271 12427 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8uoJGv/build/asan/bin/kudu as pid 27183
Sep 26 15:12:13 dist-test-slave-865v krb5kdc[26274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435533, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.195@KRBTEST.COM for kudu/127.12.34.254@KRBTEST.COM
I20260926 15:12:13.783080 27050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:13.763816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.195:36639 (local address 127.12.34.254:43645)
0926 15:12:13.764041 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0926 15:12:13.764048 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:12:13.765216 (+ 1168us) server_negotiation.cc:407] Connection header received
0926 15:12:13.766964 (+ 1748us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:12:13.766968 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:12:13.767056 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:12:13.767147 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:12:13.768594 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:12:13.769597 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:12:13.771417 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:12:13.772078 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:12:13.772173 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:12:13.776358 (+ 4185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:12:13.776363 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:12:13.776366 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:12:13.776394 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:12:13.779363 (+ 2969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:12:13.780306 (+ 943us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:12:13.780313 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:12:13.780318 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:12:13.780499 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:12:13.780996 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:12:13.781001 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:12:13.781004 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:12:13.781405 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:12:13.781567 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0926 15:12:13.782408 (+ 841us) server_negotiation.cc:299] Negotiation successful
0926 15:12:13.782671 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 15:12:13.784224 27313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:13.764342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:43645 (local address 127.12.34.195:36639)
0926 15:12:13.765017 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0926 15:12:13.765061 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 15:12:13.766698 (+ 1637us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:12:13.767344 (+ 646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:12:13.767351 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:12:13.767845 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:12:13.768412 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:12:13.768441 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:12:13.769793 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:12:13.769801 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:12:13.771266 (+ 1465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:12:13.771278 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:12:13.772461 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:12:13.772471 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:12:13.773017 (+ 546us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:12:13.773828 (+ 811us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:12:13.773852 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:12:13.776154 (+ 2302us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:12:13.779539 (+ 3385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:12:13.779547 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:12:13.779560 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:12:13.780144 (+ 584us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:12:13.780644 (+ 500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:12:13.780647 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:12:13.780650 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:12:13.780841 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:12:13.781552 (+ 711us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:12:13.781558 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:12:13.781925 (+ 367us) client_negotiation.cc:769] Sending connection context
0926 15:12:13.782287 (+ 362us) client_negotiation.cc:240] Negotiation successful
0926 15:12:13.782762 (+ 475us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":274,"threads_started":1}
I20260926 15:12:13.784987 27314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:13.769264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44664 (local address 127.12.34.254:43645)
0926 15:12:13.769781 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0926 15:12:13.769787 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:12:13.770101 (+ 314us) server_negotiation.cc:407] Connection header received
0926 15:12:13.770222 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:12:13.770226 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:12:13.770293 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:12:13.770399 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:12:13.772184 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:12:13.773548 (+ 1364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:12:13.775048 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:12:13.775548 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:12:13.775672 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:12:13.778448 (+ 2776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:12:13.778459 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:12:13.778464 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:12:13.778509 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:12:13.781728 (+ 3219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:12:13.782557 (+ 829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:12:13.782566 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:12:13.782572 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:12:13.782762 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:12:13.783324 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:12:13.783329 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:12:13.783332 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:12:13.783667 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:12:13.783962 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0926 15:12:13.784255 (+ 293us) server_negotiation.cc:299] Negotiation successful
0926 15:12:13.784514 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":397,"thread_start_us":255,"threads_started":1}
I20260926 15:12:13.786136 27311 heartbeater.cc:347] Connected to a master server at 127.12.34.254:43645
I20260926 15:12:13.786442 27311 heartbeater.cc:464] Registering TS with master...
I20260926 15:12:13.787230 27311 heartbeater.cc:511] Master 127.12.34.254:43645 requested a full tablet report, sending...
I20260926 15:12:13.788571 26862 ts_manager.cc:194] Registered new tserver with Master: b440b82da2b24d09ac47aceedd225013 (127.12.34.195:35357)
I20260926 15:12:13.789664 26862 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.34.195@KRBTEST.COM'} at 127.12.34.195:36639
I20260926 15:12:13.799813 12427 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 15:12:13.806466 27314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:13.800675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44668 (local address 127.12.34.254:43645)
0926 15:12:13.800919 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0926 15:12:13.800928 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 15:12:13.801289 (+ 361us) server_negotiation.cc:407] Connection header received
0926 15:12:13.801469 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:12:13.801472 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:12:13.801594 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:12:13.801722 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 15:12:13.802092 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:12:13.802775 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:12:13.804426 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:12:13.804843 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:12:13.804942 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:12:13.805465 (+ 523us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 15:12:13.805926 (+ 461us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 15:12:13.806036 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0926 15:12:13.806116 (+ 80us) server_negotiation.cc:299] Negotiation successful
0926 15:12:13.806181 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 15:12:13.807324 26862 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:44668:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 15:12:13.814383 26862 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 15:12:13.841167 27322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:13.825657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43752 (local address 127.12.34.193:35363)
0926 15:12:13.827651 (+ 1994us) server_negotiation.cc:206] Beginning negotiation
0926 15:12:13.827663 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 15:12:13.827697 (+ 34us) server_negotiation.cc:407] Connection header received
0926 15:12:13.827808 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:12:13.827819 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:12:13.828112 (+ 293us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:12:13.830372 (+ 2260us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:12:13.830654 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:12:13.832640 (+ 1986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:12:13.838542 (+ 5902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:12:13.839854 (+ 1312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:12:13.840002 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:12:13.840227 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0926 15:12:13.840635 (+ 408us) server_negotiation.cc:299] Negotiation successful
0926 15:12:13.840801 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1791,"thread_start_us":1620,"threads_started":1}
I20260926 15:12:13.841205 27321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:13.825370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.193:35363 (local address 127.0.0.1:43752)
0926 15:12:13.826737 (+ 1367us) negotiation.cc:107] Waiting for socket to connect
0926 15:12:13.826777 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 15:12:13.826955 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:12:13.828390 (+ 1435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:12:13.828400 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:12:13.828440 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:12:13.828957 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:12:13.828972 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:12:13.834052 (+ 5080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:12:13.834060 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:12:13.837094 (+ 3034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:12:13.837117 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:12:13.840090 (+ 2973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:12:13.840104 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:12:13.840412 (+ 308us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:12:13.840453 (+ 41us) client_negotiation.cc:769] Sending connection context
0926 15:12:13.840596 (+ 143us) client_negotiation.cc:240] Negotiation successful
0926 15:12:13.840783 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":636,"mutex_wait_us":3,"thread_start_us":509,"threads_started":1}
I20260926 15:12:13.845000 26983 tablet_service.cc:1467] Processing CreateTablet for tablet d164eccab1a9407d8cd8b6acc5aad66b (DEFAULT_TABLE table=test-table [id=ce96dfe893fd448e8fd4fd6b54ec42f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:12:13.846405 26983 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d164eccab1a9407d8cd8b6acc5aad66b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:12:13.846778 27326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:13.828681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57366 (local address 127.12.34.194:37165)
0926 15:12:13.838523 (+ 9842us) server_negotiation.cc:206] Beginning negotiation
0926 15:12:13.838534 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 15:12:13.839113 (+ 579us) server_negotiation.cc:407] Connection header received
0926 15:12:13.839215 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:12:13.839222 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:12:13.839451 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:12:13.839661 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:12:13.840740 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:12:13.842281 (+ 1541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:12:13.844407 (+ 2126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:12:13.845701 (+ 1294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:12:13.845838 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:12:13.846012 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0926 15:12:13.846295 (+ 283us) server_negotiation.cc:299] Negotiation successful
0926 15:12:13.846472 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9686,"thread_start_us":9573,"threads_started":1}
I20260926 15:12:13.847126 27325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:13.827998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.194:37165 (local address 127.0.0.1:57366)
0926 15:12:13.838845 (+ 10847us) negotiation.cc:107] Waiting for socket to connect
0926 15:12:13.838864 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 15:12:13.838987 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:12:13.839666 (+ 679us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:12:13.839671 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:12:13.839697 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:12:13.840546 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:12:13.840562 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:12:13.842436 (+ 1874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:12:13.842443 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:12:13.844268 (+ 1825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:12:13.844283 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:12:13.845916 (+ 1633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:12:13.845922 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:12:13.846105 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:12:13.846135 (+ 30us) client_negotiation.cc:769] Sending connection context
0926 15:12:13.846237 (+ 102us) client_negotiation.cc:240] Negotiation successful
0926 15:12:13.846355 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10730,"mutex_wait_us":48,"thread_start_us":10588,"threads_started":1}
I20260926 15:12:13.847541 27323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:13.825710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.195:35357 (local address 127.0.0.1:57620)
0926 15:12:13.827912 (+ 2202us) negotiation.cc:107] Waiting for socket to connect
0926 15:12:13.827928 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 15:12:13.828028 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:12:13.839692 (+ 11664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:12:13.839697 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:12:13.839724 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:12:13.840178 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:12:13.840191 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:12:13.842465 (+ 2274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:12:13.842471 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:12:13.844762 (+ 2291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:12:13.844776 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:12:13.846623 (+ 1847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:12:13.846632 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:12:13.846834 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:12:13.846872 (+ 38us) client_negotiation.cc:769] Sending connection context
0926 15:12:13.846956 (+ 84us) client_negotiation.cc:240] Negotiation successful
0926 15:12:13.847089 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2056,"mutex_wait_us":153,"thread_start_us":1935,"threads_started":1}
I20260926 15:12:13.848516 27324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:13.826513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57620 (local address 127.12.34.195:35357)
0926 15:12:13.839220 (+ 12707us) server_negotiation.cc:206] Beginning negotiation
0926 15:12:13.839227 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:12:13.839252 (+ 25us) server_negotiation.cc:407] Connection header received
0926 15:12:13.839333 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:12:13.839338 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:12:13.839501 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:12:13.839856 (+ 355us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:12:13.840405 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:12:13.842316 (+ 1911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:12:13.844927 (+ 2611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:12:13.846275 (+ 1348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:12:13.846519 (+ 244us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:12:13.847959 (+ 1440us) server_negotiation.cc:1035] Waiting for connection context
0926 15:12:13.848043 (+ 84us) server_negotiation.cc:299] Negotiation successful
0926 15:12:13.848188 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12548,"thread_start_us":12362,"threads_started":1}
I20260926 15:12:13.850733 27115 tablet_service.cc:1467] Processing CreateTablet for tablet d164eccab1a9407d8cd8b6acc5aad66b (DEFAULT_TABLE table=test-table [id=ce96dfe893fd448e8fd4fd6b54ec42f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:12:13.851105 27246 tablet_service.cc:1467] Processing CreateTablet for tablet d164eccab1a9407d8cd8b6acc5aad66b (DEFAULT_TABLE table=test-table [id=ce96dfe893fd448e8fd4fd6b54ec42f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:12:13.852051 27115 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d164eccab1a9407d8cd8b6acc5aad66b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:12:13.852176 27246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d164eccab1a9407d8cd8b6acc5aad66b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:12:13.871187 27328 tablet_bootstrap.cc:492] T d164eccab1a9407d8cd8b6acc5aad66b P b440b82da2b24d09ac47aceedd225013: Bootstrap starting.
I20260926 15:12:13.873688 27327 tablet_bootstrap.cc:492] T d164eccab1a9407d8cd8b6acc5aad66b P 4067cd15627b4f5ba7f5eb07551e7fab: Bootstrap starting.
I20260926 15:12:13.876159 27329 tablet_bootstrap.cc:492] T d164eccab1a9407d8cd8b6acc5aad66b P cbe51bf32d864c85a4189dc5531617b1: Bootstrap starting.
I20260926 15:12:13.877527 27327 tablet_bootstrap.cc:654] T d164eccab1a9407d8cd8b6acc5aad66b P 4067cd15627b4f5ba7f5eb07551e7fab: Neither blocks nor log segments found. Creating new log.
I20260926 15:12:13.877539 27328 tablet_bootstrap.cc:654] T d164eccab1a9407d8cd8b6acc5aad66b P b440b82da2b24d09ac47aceedd225013: Neither blocks nor log segments found. Creating new log.
I20260926 15:12:13.878750 27327 log.cc:824] T d164eccab1a9407d8cd8b6acc5aad66b P 4067cd15627b4f5ba7f5eb07551e7fab: Log is configured to *not* fsync() on all Append() calls
I20260926 15:12:13.878750 27328 log.cc:824] T d164eccab1a9407d8cd8b6acc5aad66b P b440b82da2b24d09ac47aceedd225013: Log is configured to *not* fsync() on all Append() calls
I20260926 15:12:13.878820 27329 tablet_bootstrap.cc:654] T d164eccab1a9407d8cd8b6acc5aad66b P cbe51bf32d864c85a4189dc5531617b1: Neither blocks nor log segments found. Creating new log.
I20260926 15:12:13.879994 27329 log.cc:824] T d164eccab1a9407d8cd8b6acc5aad66b P cbe51bf32d864c85a4189dc5531617b1: Log is configured to *not* fsync() on all Append() calls
I20260926 15:12:13.881776 27328 tablet_bootstrap.cc:492] T d164eccab1a9407d8cd8b6acc5aad66b P b440b82da2b24d09ac47aceedd225013: No bootstrap required, opened a new log
I20260926 15:12:13.882027 27328 ts_tablet_manager.cc:1397] T d164eccab1a9407d8cd8b6acc5aad66b P b440b82da2b24d09ac47aceedd225013: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.001s
I20260926 15:12:13.882501 27327 tablet_bootstrap.cc:492] T d164eccab1a9407d8cd8b6acc5aad66b P 4067cd15627b4f5ba7f5eb07551e7fab: No bootstrap required, opened a new log
I20260926 15:12:13.882781 27327 ts_tablet_manager.cc:1397] T d164eccab1a9407d8cd8b6acc5aad66b P 4067cd15627b4f5ba7f5eb07551e7fab: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260926 15:12:13.882891 27329 tablet_bootstrap.cc:492] T d164eccab1a9407d8cd8b6acc5aad66b P cbe51bf32d864c85a4189dc5531617b1: No bootstrap required, opened a new log
I20260926 15:12:13.883139 27329 ts_tablet_manager.cc:1397] T d164eccab1a9407d8cd8b6acc5aad66b P cbe51bf32d864c85a4189dc5531617b1: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260926 15:12:13.889760 27328 raft_consensus.cc:348] T d164eccab1a9407d8cd8b6acc5aad66b P b440b82da2b24d09ac47aceedd225013 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4067cd15627b4f5ba7f5eb07551e7fab" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 35363 } } peers { permanent_uuid: "b440b82da2b24d09ac47aceedd225013" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 35357 } } peers { permanent_uuid: "cbe51bf32d864c85a4189dc5531617b1" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 37165 } }
I20260926 15:12:13.890177 27328 raft_consensus.cc:374] T d164eccab1a9407d8cd8b6acc5aad66b P b440b82da2b24d09ac47aceedd225013 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:12:13.890278 27328 raft_consensus.cc:729] T d164eccab1a9407d8cd8b6acc5aad66b P b440b82da2b24d09ac47aceedd225013 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b440b82da2b24d09ac47aceedd225013, State: Initialized, Role: FOLLOWER
I20260926 15:12:13.890959 27328 consensus_queue.cc:260] T d164eccab1a9407d8cd8b6acc5aad66b P b440b82da2b24d09ac47aceedd225013 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4067cd15627b4f5ba7f5eb07551e7fab" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 35363 } } peers { permanent_uuid: "b440b82da2b24d09ac47aceedd225013" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 35357 } } peers { permanent_uuid: "cbe51bf32d864c85a4189dc5531617b1" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 37165 } }
I20260926 15:12:13.892177 27311 heartbeater.cc:503] Master 127.12.34.254:43645 was elected leader, sending a full tablet report...
I20260926 15:12:13.892859 27328 ts_tablet_manager.cc:1428] T d164eccab1a9407d8cd8b6acc5aad66b P b440b82da2b24d09ac47aceedd225013: Time spent starting tablet: real 0.011s user 0.003s sys 0.008s
I20260926 15:12:13.893613 27327 raft_consensus.cc:348] T d164eccab1a9407d8cd8b6acc5aad66b P 4067cd15627b4f5ba7f5eb07551e7fab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4067cd15627b4f5ba7f5eb07551e7fab" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 35363 } } peers { permanent_uuid: "b440b82da2b24d09ac47aceedd225013" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 35357 } } peers { permanent_uuid: "cbe51bf32d864c85a4189dc5531617b1" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 37165 } }
I20260926 15:12:13.893579 27329 raft_consensus.cc:348] T d164eccab1a9407d8cd8b6acc5aad66b P cbe51bf32d864c85a4189dc5531617b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4067cd15627b4f5ba7f5eb07551e7fab" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 35363 } } peers { permanent_uuid: "b440b82da2b24d09ac47aceedd225013" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 35357 } } peers { permanent_uuid: "cbe51bf32d864c85a4189dc5531617b1" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 37165 } }
I20260926 15:12:13.893996 27327 raft_consensus.cc:374] T d164eccab1a9407d8cd8b6acc5aad66b P 4067cd15627b4f5ba7f5eb07551e7fab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:12:13.893996 27329 raft_consensus.cc:374] T d164eccab1a9407d8cd8b6acc5aad66b P cbe51bf32d864c85a4189dc5531617b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:12:13.894102 27329 raft_consensus.cc:729] T d164eccab1a9407d8cd8b6acc5aad66b P cbe51bf32d864c85a4189dc5531617b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbe51bf32d864c85a4189dc5531617b1, State: Initialized, Role: FOLLOWER
I20260926 15:12:13.894102 27327 raft_consensus.cc:729] T d164eccab1a9407d8cd8b6acc5aad66b P 4067cd15627b4f5ba7f5eb07551e7fab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4067cd15627b4f5ba7f5eb07551e7fab, State: Initialized, Role: FOLLOWER
I20260926 15:12:13.894701 27327 consensus_queue.cc:260] T d164eccab1a9407d8cd8b6acc5aad66b P 4067cd15627b4f5ba7f5eb07551e7fab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4067cd15627b4f5ba7f5eb07551e7fab" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 35363 } } peers { permanent_uuid: "b440b82da2b24d09ac47aceedd225013" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 35357 } } peers { permanent_uuid: "cbe51bf32d864c85a4189dc5531617b1" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 37165 } }
I20260926 15:12:13.894666 27329 consensus_queue.cc:260] T d164eccab1a9407d8cd8b6acc5aad66b P cbe51bf32d864c85a4189dc5531617b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4067cd15627b4f5ba7f5eb07551e7fab" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 35363 } } peers { permanent_uuid: "b440b82da2b24d09ac47aceedd225013" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 35357 } } peers { permanent_uuid: "cbe51bf32d864c85a4189dc5531617b1" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 37165 } }
I20260926 15:12:13.896406 27329 ts_tablet_manager.cc:1428] T d164eccab1a9407d8cd8b6acc5aad66b P cbe51bf32d864c85a4189dc5531617b1: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
I20260926 15:12:13.896703 27327 ts_tablet_manager.cc:1428] T d164eccab1a9407d8cd8b6acc5aad66b P 4067cd15627b4f5ba7f5eb07551e7fab: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260926 15:12:13.896972 27048 heartbeater.cc:503] Master 127.12.34.254:43645 was elected leader, sending a full tablet report...
I20260926 15:12:13.897212 27180 heartbeater.cc:503] Master 127.12.34.254:43645 was elected leader, sending a full tablet report...
I20260926 15:12:13.963150 27335 raft_consensus.cc:482] T d164eccab1a9407d8cd8b6acc5aad66b P 4067cd15627b4f5ba7f5eb07551e7fab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 15:12:13.963492 27335 raft_consensus.cc:504] T d164eccab1a9407d8cd8b6acc5aad66b P 4067cd15627b4f5ba7f5eb07551e7fab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4067cd15627b4f5ba7f5eb07551e7fab" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 35363 } } peers { permanent_uuid: "b440b82da2b24d09ac47aceedd225013" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 35357 } } peers { permanent_uuid: "cbe51bf32d864c85a4189dc5531617b1" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 37165 } }
I20260926 15:12:13.965255 27335 leader_election.cc:288] T d164eccab1a9407d8cd8b6acc5aad66b P 4067cd15627b4f5ba7f5eb07551e7fab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b440b82da2b24d09ac47aceedd225013 (127.12.34.195:35357), cbe51bf32d864c85a4189dc5531617b1 (127.12.34.194:37165)
I20260926 15:12:13.973455 27326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:13.966490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.193:47385 (local address 127.12.34.194:37165)
0926 15:12:13.966719 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0926 15:12:13.966725 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:12:13.967108 (+ 383us) server_negotiation.cc:407] Connection header received
0926 15:12:13.967260 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:12:13.967264 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:12:13.967320 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:12:13.967425 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:12:13.967867 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:12:13.968777 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:12:13.971145 (+ 2368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:12:13.972383 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:12:13.972513 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:12:13.972581 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 15:12:13.973184 (+ 603us) server_negotiation.cc:299] Negotiation successful
0926 15:12:13.973272 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260926 15:12:13.973791 27337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:13.966342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.194:37165 (local address 127.12.34.193:47385)
0926 15:12:13.966999 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0926 15:12:13.967020 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 15:12:13.967144 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:12:13.967455 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:12:13.967458 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:12:13.967477 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:12:13.967728 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:12:13.967735 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:12:13.968935 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:12:13.968940 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:12:13.970966 (+ 2026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:12:13.970977 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:12:13.972628 (+ 1651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:12:13.972632 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:12:13.972957 (+ 325us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:12:13.973048 (+ 91us) client_negotiation.cc:769] Sending connection context
0926 15:12:13.973164 (+ 116us) client_negotiation.cc:240] Negotiation successful
0926 15:12:13.973320 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"mutex_wait_us":126,"thread_start_us":302,"threads_started":1}
I20260926 15:12:13.973816 27336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:13.965564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.195:35357 (local address 127.12.34.193:35731)
0926 15:12:13.966185 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0926 15:12:13.966200 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 15:12:13.966369 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:12:13.966835 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:12:13.966840 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:12:13.966866 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:12:13.967282 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:12:13.967292 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:12:13.968567 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:12:13.968571 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:12:13.970851 (+ 2280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:12:13.970872 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:12:13.972623 (+ 1751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:12:13.972630 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:12:13.972954 (+ 324us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:12:13.973053 (+ 99us) client_negotiation.cc:769] Sending connection context
0926 15:12:13.973164 (+ 111us) client_negotiation.cc:240] Negotiation successful
0926 15:12:13.973320 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"mutex_wait_us":136,"thread_start_us":350,"threads_started":1}
I20260926 15:12:13.974140 27324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:13.965678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.193:35731 (local address 127.12.34.195:35357)
0926 15:12:13.965935 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0926 15:12:13.965942 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:12:13.966301 (+ 359us) server_negotiation.cc:407] Connection header received
0926 15:12:13.966543 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:12:13.966551 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:12:13.966659 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:12:13.966757 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:12:13.967423 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:12:13.968431 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:12:13.971025 (+ 2594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:12:13.972337 (+ 1312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:12:13.972504 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:12:13.972586 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 15:12:13.973734 (+ 1148us) server_negotiation.cc:299] Negotiation successful
0926 15:12:13.973836 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 15:12:13.975171 27135 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d164eccab1a9407d8cd8b6acc5aad66b" candidate_uuid: "4067cd15627b4f5ba7f5eb07551e7fab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cbe51bf32d864c85a4189dc5531617b1" is_pre_election: true
I20260926 15:12:13.975267 27266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d164eccab1a9407d8cd8b6acc5aad66b" candidate_uuid: "4067cd15627b4f5ba7f5eb07551e7fab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b440b82da2b24d09ac47aceedd225013" is_pre_election: true
I20260926 15:12:13.975679 27135 raft_consensus.cc:2454] T d164eccab1a9407d8cd8b6acc5aad66b P cbe51bf32d864c85a4189dc5531617b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4067cd15627b4f5ba7f5eb07551e7fab in term 0.
I20260926 15:12:13.975736 27266 raft_consensus.cc:2454] T d164eccab1a9407d8cd8b6acc5aad66b P b440b82da2b24d09ac47aceedd225013 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4067cd15627b4f5ba7f5eb07551e7fab in term 0.
I20260926 15:12:13.976436 26936 leader_election.cc:302] T d164eccab1a9407d8cd8b6acc5aad66b P 4067cd15627b4f5ba7f5eb07551e7fab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4067cd15627b4f5ba7f5eb07551e7fab, cbe51bf32d864c85a4189dc5531617b1; no voters:
I20260926 15:12:13.976830 27335 raft_consensus.cc:2781] T d164eccab1a9407d8cd8b6acc5aad66b P 4067cd15627b4f5ba7f5eb07551e7fab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 15:12:13.976979 27335 raft_consensus.cc:482] T d164eccab1a9407d8cd8b6acc5aad66b P 4067cd15627b4f5ba7f5eb07551e7fab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 15:12:13.977104 27335 raft_consensus.cc:3037] T d164eccab1a9407d8cd8b6acc5aad66b P 4067cd15627b4f5ba7f5eb07551e7fab [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:12:13.979384 27335 raft_consensus.cc:504] T d164eccab1a9407d8cd8b6acc5aad66b P 4067cd15627b4f5ba7f5eb07551e7fab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4067cd15627b4f5ba7f5eb07551e7fab" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 35363 } } peers { permanent_uuid: "b440b82da2b24d09ac47aceedd225013" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 35357 } } peers { permanent_uuid: "cbe51bf32d864c85a4189dc5531617b1" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 37165 } }
I20260926 15:12:13.979969 27335 leader_election.cc:288] T d164eccab1a9407d8cd8b6acc5aad66b P 4067cd15627b4f5ba7f5eb07551e7fab [CANDIDATE]: Term 1 election: Requested vote from peers b440b82da2b24d09ac47aceedd225013 (127.12.34.195:35357), cbe51bf32d864c85a4189dc5531617b1 (127.12.34.194:37165)
I20260926 15:12:13.980515 27266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d164eccab1a9407d8cd8b6acc5aad66b" candidate_uuid: "4067cd15627b4f5ba7f5eb07551e7fab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b440b82da2b24d09ac47aceedd225013"
I20260926 15:12:13.980515 27135 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d164eccab1a9407d8cd8b6acc5aad66b" candidate_uuid: "4067cd15627b4f5ba7f5eb07551e7fab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cbe51bf32d864c85a4189dc5531617b1"
I20260926 15:12:13.980731 27266 raft_consensus.cc:3037] T d164eccab1a9407d8cd8b6acc5aad66b P b440b82da2b24d09ac47aceedd225013 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:12:13.980731 27135 raft_consensus.cc:3037] T d164eccab1a9407d8cd8b6acc5aad66b P cbe51bf32d864c85a4189dc5531617b1 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:12:13.983968 27135 raft_consensus.cc:2454] T d164eccab1a9407d8cd8b6acc5aad66b P cbe51bf32d864c85a4189dc5531617b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4067cd15627b4f5ba7f5eb07551e7fab in term 1.
I20260926 15:12:13.984010 27266 raft_consensus.cc:2454] T d164eccab1a9407d8cd8b6acc5aad66b P b440b82da2b24d09ac47aceedd225013 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4067cd15627b4f5ba7f5eb07551e7fab in term 1.
I20260926 15:12:13.984417 26936 leader_election.cc:302] T d164eccab1a9407d8cd8b6acc5aad66b P 4067cd15627b4f5ba7f5eb07551e7fab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4067cd15627b4f5ba7f5eb07551e7fab, cbe51bf32d864c85a4189dc5531617b1; no voters:
I20260926 15:12:13.984679 27335 raft_consensus.cc:2781] T d164eccab1a9407d8cd8b6acc5aad66b P 4067cd15627b4f5ba7f5eb07551e7fab [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:12:13.985278 27335 raft_consensus.cc:686] T d164eccab1a9407d8cd8b6acc5aad66b P 4067cd15627b4f5ba7f5eb07551e7fab [term 1 LEADER]: Becoming Leader. State: Replica: 4067cd15627b4f5ba7f5eb07551e7fab, State: Running, Role: LEADER
I20260926 15:12:13.985785 27335 consensus_queue.cc:237] T d164eccab1a9407d8cd8b6acc5aad66b P 4067cd15627b4f5ba7f5eb07551e7fab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4067cd15627b4f5ba7f5eb07551e7fab" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 35363 } } peers { permanent_uuid: "b440b82da2b24d09ac47aceedd225013" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 35357 } } peers { permanent_uuid: "cbe51bf32d864c85a4189dc5531617b1" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 37165 } }
I20260926 15:12:13.991276 26862 catalog_manager.cc:5495] T d164eccab1a9407d8cd8b6acc5aad66b P 4067cd15627b4f5ba7f5eb07551e7fab reported cstate change: term changed from 0 to 1, leader changed from <none> to 4067cd15627b4f5ba7f5eb07551e7fab (127.12.34.193). New cstate: current_term: 1 leader_uuid: "4067cd15627b4f5ba7f5eb07551e7fab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4067cd15627b4f5ba7f5eb07551e7fab" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 35363 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b440b82da2b24d09ac47aceedd225013" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 35357 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cbe51bf32d864c85a4189dc5531617b1" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 37165 } health_report { overall_health: UNKNOWN } } }
W20260926 15:12:14.012894 27312 tablet.cc:2369] T d164eccab1a9407d8cd8b6acc5aad66b P b440b82da2b24d09ac47aceedd225013: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 15:12:14.026889 27322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:14.021345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43760 (local address 127.12.34.193:35363)
0926 15:12:14.021578 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0926 15:12:14.021586 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 15:12:14.021702 (+ 116us) server_negotiation.cc:407] Connection header received
0926 15:12:14.021979 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:12:14.021984 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:12:14.022077 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:12:14.022228 (+ 151us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 15:12:14.022597 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:12:14.023323 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:12:14.024713 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:12:14.025138 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:12:14.025273 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:12:14.025741 (+ 468us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 15:12:14.026227 (+ 486us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 15:12:14.026398 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0926 15:12:14.026454 (+ 56us) server_negotiation.cc:299] Negotiation successful
0926 15:12:14.026533 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260926 15:12:14.034852 27266 raft_consensus.cc:1261] T d164eccab1a9407d8cd8b6acc5aad66b P b440b82da2b24d09ac47aceedd225013 [term 1 FOLLOWER]: Refusing update from remote peer 4067cd15627b4f5ba7f5eb07551e7fab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:12:14.034849 27135 raft_consensus.cc:1261] T d164eccab1a9407d8cd8b6acc5aad66b P cbe51bf32d864c85a4189dc5531617b1 [term 1 FOLLOWER]: Refusing update from remote peer 4067cd15627b4f5ba7f5eb07551e7fab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:12:14.035782 27338 consensus_queue.cc:1035] T d164eccab1a9407d8cd8b6acc5aad66b P 4067cd15627b4f5ba7f5eb07551e7fab [LEADER]: Connected to new peer: Peer: permanent_uuid: "b440b82da2b24d09ac47aceedd225013" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 35357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:12:14.036159 27335 consensus_queue.cc:1035] T d164eccab1a9407d8cd8b6acc5aad66b P 4067cd15627b4f5ba7f5eb07551e7fab [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbe51bf32d864c85a4189dc5531617b1" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 37165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:12:14.046442 27343 mvcc.cc:205] Tried to move back new op lower bound from 7333623947397185536 to 7333623947209818112. Current Snapshot: MvccSnapshot[applied={T|T < 7333623947397185536}]
I20260926 15:12:14.060508 26861 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:44668:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 15:12:14.060786 26861 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:44668:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 15:12:14.064581 26861 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2e76b19894564aa28e81f161acd96f87: Sending DeleteTablet for 3 replicas of tablet d164eccab1a9407d8cd8b6acc5aad66b
I20260926 15:12:14.065737 26983 tablet_service.cc:1514] Processing DeleteTablet for tablet d164eccab1a9407d8cd8b6acc5aad66b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:12:14 UTC) from {username='kudu', principal='kudu/127.12.34.254@KRBTEST.COM'} at 127.0.0.1:43752
I20260926 15:12:14.065951 12427 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 26899
I20260926 15:12:14.065953 27246 tablet_service.cc:1514] Processing DeleteTablet for tablet d164eccab1a9407d8cd8b6acc5aad66b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:12:14 UTC) from {username='kudu', principal='kudu/127.12.34.254@KRBTEST.COM'} at 127.0.0.1:57620
I20260926 15:12:14.066720 27115 tablet_service.cc:1514] Processing DeleteTablet for tablet d164eccab1a9407d8cd8b6acc5aad66b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:12:14 UTC) from {username='kudu', principal='kudu/127.12.34.254@KRBTEST.COM'} at 127.0.0.1:57366
I20260926 15:12:14.066905 27351 tablet_replica.cc:330] T d164eccab1a9407d8cd8b6acc5aad66b P b440b82da2b24d09ac47aceedd225013: stopping tablet replica
I20260926 15:12:14.067348 27351 raft_consensus.cc:2229] T d164eccab1a9407d8cd8b6acc5aad66b P b440b82da2b24d09ac47aceedd225013 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 15:12:14.067829 27351 raft_consensus.cc:2258] T d164eccab1a9407d8cd8b6acc5aad66b P b440b82da2b24d09ac47aceedd225013 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:12:14.068634 27352 tablet_replica.cc:330] T d164eccab1a9407d8cd8b6acc5aad66b P cbe51bf32d864c85a4189dc5531617b1: stopping tablet replica
I20260926 15:12:14.069072 27352 raft_consensus.cc:2229] T d164eccab1a9407d8cd8b6acc5aad66b P cbe51bf32d864c85a4189dc5531617b1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 15:12:14.069638 27352 raft_consensus.cc:2258] T d164eccab1a9407d8cd8b6acc5aad66b P cbe51bf32d864c85a4189dc5531617b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:12:14.070160 27351 ts_tablet_manager.cc:1905] T d164eccab1a9407d8cd8b6acc5aad66b P b440b82da2b24d09ac47aceedd225013: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:12:14.074198 27352 ts_tablet_manager.cc:1905] T d164eccab1a9407d8cd8b6acc5aad66b P cbe51bf32d864c85a4189dc5531617b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:12:14.078006 27351 ts_tablet_manager.cc:1918] T d164eccab1a9407d8cd8b6acc5aad66b P b440b82da2b24d09ac47aceedd225013: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 15:12:14.078220 27351 log.cc:1196] T d164eccab1a9407d8cd8b6acc5aad66b P b440b82da2b24d09ac47aceedd225013: Deleting WAL directory at /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/ts-2/wal/wals/d164eccab1a9407d8cd8b6acc5aad66b
I20260926 15:12:14.078876 27351 ts_tablet_manager.cc:1939] T d164eccab1a9407d8cd8b6acc5aad66b P b440b82da2b24d09ac47aceedd225013: Deleting consensus metadata
I20260926 15:12:14.080981 26846 catalog_manager.cc:4841] TS b440b82da2b24d09ac47aceedd225013 (127.12.34.195:35357): tablet d164eccab1a9407d8cd8b6acc5aad66b (table test-table [id=ce96dfe893fd448e8fd4fd6b54ec42f2]) successfully deleted
I20260926 15:12:14.081965 27350 tablet_replica.cc:330] T d164eccab1a9407d8cd8b6acc5aad66b P 4067cd15627b4f5ba7f5eb07551e7fab: stopping tablet replica
I20260926 15:12:14.082386 27350 raft_consensus.cc:2229] T d164eccab1a9407d8cd8b6acc5aad66b P 4067cd15627b4f5ba7f5eb07551e7fab [term 1 LEADER]: Raft consensus shutting down.
I20260926 15:12:14.083076 27350 raft_consensus.cc:2258] T d164eccab1a9407d8cd8b6acc5aad66b P 4067cd15627b4f5ba7f5eb07551e7fab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:12:14.084748 27322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:14.067637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43770 (local address 127.12.34.193:35363)
0926 15:12:14.067855 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0926 15:12:14.067864 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 15:12:14.067888 (+ 24us) server_negotiation.cc:407] Connection header received
0926 15:12:14.068091 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:12:14.068097 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:12:14.068174 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:12:14.068268 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:12:14.069996 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:12:14.071266 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:12:14.073073 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:12:14.073767 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:12:14.073918 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:12:14.077171 (+ 3253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:12:14.077184 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:12:14.077201 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:12:14.077253 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:12:14.081346 (+ 4093us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:12:14.082197 (+ 851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:12:14.082206 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:12:14.082211 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:12:14.082389 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:12:14.082859 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:12:14.082865 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:12:14.082869 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:12:14.083353 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:12:14.083851 (+ 498us) server_negotiation.cc:1035] Waiting for connection context
0926 15:12:14.083997 (+ 146us) server_negotiation.cc:299] Negotiation successful
0926 15:12:14.084217 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260926 15:12:14.085712 27043 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:12:14.087507 27352 ts_tablet_manager.cc:1918] T d164eccab1a9407d8cd8b6acc5aad66b P cbe51bf32d864c85a4189dc5531617b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 15:12:14.087651 27352 log.cc:1196] T d164eccab1a9407d8cd8b6acc5aad66b P cbe51bf32d864c85a4189dc5531617b1: Deleting WAL directory at /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/ts-1/wal/wals/d164eccab1a9407d8cd8b6acc5aad66b
I20260926 15:12:14.088171 27352 ts_tablet_manager.cc:1939] T d164eccab1a9407d8cd8b6acc5aad66b P cbe51bf32d864c85a4189dc5531617b1: Deleting consensus metadata
I20260926 15:12:14.089795 26846 catalog_manager.cc:4841] TS cbe51bf32d864c85a4189dc5531617b1 (127.12.34.194:37165): tablet d164eccab1a9407d8cd8b6acc5aad66b (table test-table [id=ce96dfe893fd448e8fd4fd6b54ec42f2]) successfully deleted
I20260926 15:12:14.086246 27350 ts_tablet_manager.cc:1905] T d164eccab1a9407d8cd8b6acc5aad66b P 4067cd15627b4f5ba7f5eb07551e7fab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:12:14.184098 12427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8uoJGv/build/asan/bin/kudu with pid 26899
W20260926 15:12:14.202347 26848 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.34.193:35363 (error 108)
I20260926 15:12:14.202544 12427 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 27052
I20260926 15:12:14.204617 27323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:14.204119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.193:35363 (local address 127.0.0.1:43772)
0926 15:12:14.204329 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0926 15:12:14.204446 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.193:35363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260926 15:12:14.204885 26848 catalog_manager.cc:4584] TS 4067cd15627b4f5ba7f5eb07551e7fab (127.12.34.193:35363): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d164eccab1a9407d8cd8b6acc5aad66b: Network error: Client connection negotiation failed: client connection to 127.12.34.193:35363: connect: Connection refused (error 111)
I20260926 15:12:14.216135 27326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:14.203370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57370 (local address 127.12.34.194:37165)
0926 15:12:14.203587 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0926 15:12:14.203594 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:12:14.203821 (+ 227us) server_negotiation.cc:407] Connection header received
0926 15:12:14.204036 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:12:14.204042 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:12:14.204134 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:12:14.204330 (+ 196us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:12:14.205794 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:12:14.206512 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:12:14.207777 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:12:14.208161 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:12:14.208252 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:12:14.210669 (+ 2417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:12:14.210681 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:12:14.210694 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:12:14.210723 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:12:14.213532 (+ 2809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:12:14.214138 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:12:14.214144 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:12:14.214147 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:12:14.214257 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:12:14.214561 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:12:14.214564 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:12:14.214566 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:12:14.214912 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:12:14.215298 (+ 386us) server_negotiation.cc:1035] Waiting for connection context
0926 15:12:14.215530 (+ 232us) server_negotiation.cc:299] Negotiation successful
0926 15:12:14.215709 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 15:12:14.216823 27175 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:12:14.258380 27323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:14.257907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.193:35363 (local address 127.0.0.1:43776)
0926 15:12:14.258114 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0926 15:12:14.258219 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.193:35363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260926 15:12:14.259536 27323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:14.259152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.193:35363 (local address 127.0.0.1:43782)
0926 15:12:14.259321 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0926 15:12:14.259420 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.193:35363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260926 15:12:14.312800 12427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8uoJGv/build/asan/bin/kudu with pid 27052
I20260926 15:12:14.320799 27323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:14.320085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.193:35363 (local address 127.0.0.1:43784)
0926 15:12:14.320426 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0926 15:12:14.320562 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.193:35363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185}
I20260926 15:12:14.322297 27323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:14.321828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.193:35363 (local address 127.0.0.1:43792)
0926 15:12:14.322041 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0926 15:12:14.322131 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.193:35363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260926 15:12:14.330363 12427 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 27183
I20260926 15:12:14.345589 27324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:14.331158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57634 (local address 127.12.34.195:35357)
0926 15:12:14.331402 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0926 15:12:14.331408 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:12:14.331429 (+ 21us) server_negotiation.cc:407] Connection header received
0926 15:12:14.331494 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:12:14.331500 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:12:14.331580 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:12:14.331706 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:12:14.333007 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:12:14.334188 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:12:14.335578 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:12:14.336185 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:12:14.336327 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:12:14.339127 (+ 2800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:12:14.339144 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:12:14.339166 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:12:14.339207 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:12:14.342574 (+ 3367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:12:14.343403 (+ 829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:12:14.343414 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:12:14.343417 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:12:14.343753 (+ 336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:12:14.344169 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:12:14.344177 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:12:14.344179 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:12:14.344567 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:12:14.344798 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0926 15:12:14.345159 (+ 361us) server_negotiation.cc:299] Negotiation successful
0926 15:12:14.345340 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 15:12:14.346252 27306 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:12:14.403692 27323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:14.403196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.193:35363 (local address 127.0.0.1:43794)
0926 15:12:14.403428 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0926 15:12:14.403520 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.193:35363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260926 15:12:14.404793 27323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:14.404400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.193:35363 (local address 127.0.0.1:43798)
0926 15:12:14.404617 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0926 15:12:14.404687 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.193:35363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123}
I20260926 15:12:14.438845 12427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8uoJGv/build/asan/bin/kudu with pid 27183
I20260926 15:12:14.455153 12427 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 26829
I20260926 15:12:14.456288 26892 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:12:14.570900 12427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8uoJGv/build/asan/bin/kudu with pid 26829
I20260926 15:12:14.592434 12427 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8uoJGv/build/asan/bin/kudu
/tmp/dist-test-task8uoJGv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.34.254:43645
--webserver_interface=127.12.34.254
--webserver_port=42385
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.254.keytab
--principal=kudu/127.12.34.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.34.212:36007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.34.254:43645
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:12:14.922605 27358 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:12:14.922942 27358 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:12:14.923110 27358 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:12:14.930526 27358 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:12:14.930668 27358 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:12:14.930765 27358 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:12:14.930866 27358 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:12:14.930938 27358 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:12:14.940698 27358 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.34.212:36007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.34.254:43645
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.254.keytab
--principal=kudu/127.12.34.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.34.254:43645
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.34.254
--webserver_port=42385
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:12:14.942082 27358 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:12:14.943357 27358 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:12:14.954829 27363 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:12:14.954807 27364 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:12:14.955003 27366 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:12:14.956198 27358 server_base.cc:1034] running on GCE node
I20260926 15:12:14.956933 27358 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:12:14.958356 27358 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:12:14.959589 27358 hybrid_clock.cc:648] HybridClock initialized: now 1790435534959548 us; error 52 us; skew 500 ppm
I20260926 15:12:14.959909 27358 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:12:14 dist-test-slave-865v krb5kdc[26274](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435534, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:12:14.962854 27358 init.cc:377] Logged in from keytab as kudu/127.12.34.254@KRBTEST.COM (short username kudu)
I20260926 15:12:14.964378 27358 webserver.cc:460] Webserver started at http://127.12.34.254:42385/ using document root <none> and password file <none>
I20260926 15:12:14.965070 27358 fs_manager.cc:359] Metadata directory not provided
I20260926 15:12:14.965260 27358 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:12:14.970332 27358 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260926 15:12:14.973145 27373 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:12:14.974478 27358 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 15:12:14.974658 27358 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/master-0/data,/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/master-0/wal
uuid: "2e76b19894564aa28e81f161acd96f87"
format_stamp: "Formatted at 2026-09-26 15:12:09 on dist-test-slave-865v"
server_key: "4cd98550f90faf5dce91d40b3617c5c3"
server_key_iv: "5bc885476fa2d48422d0345de0ed7477"
server_key_version: "encryptionkey@0"
I20260926 15:12:14.975607 27358 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:12:15.002067 27358 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:12:15.006337 27358 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:12:15.006608 27358 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:12:15.026345 27358 rpc_server.cc:307] RPC server started. Bound to: 127.12.34.254:43645
I20260926 15:12:15.026372 27424 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.34.254:43645 every 8 connection(s)
I20260926 15:12:15.027854 27358 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
I20260926 15:12:15.034775 27425 sys_catalog.cc:263] Verifying existing consensus state
I20260926 15:12:15.035097 12427 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8uoJGv/build/asan/bin/kudu as pid 27358
I20260926 15:12:15.036082 12427 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8uoJGv/build/asan/bin/kudu
/tmp/dist-test-task8uoJGv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.34.193:35363
--local_ip_for_outbound_sockets=127.12.34.193
--tserver_master_addrs=127.12.34.254:43645
--webserver_port=33943
--webserver_interface=127.12.34.193
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.193.keytab
--principal=kudu/127.12.34.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.34.212:36007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 15:12:15.039759 27425 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e76b19894564aa28e81f161acd96f87: Bootstrap starting.
I20260926 15:12:15.061184 27425 log.cc:824] T 00000000000000000000000000000000 P 2e76b19894564aa28e81f161acd96f87: Log is configured to *not* fsync() on all Append() calls
I20260926 15:12:15.076954 27425 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e76b19894564aa28e81f161acd96f87: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 15:12:15.077548 27425 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e76b19894564aa28e81f161acd96f87: Bootstrap complete.
I20260926 15:12:15.087455 27425 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2e76b19894564aa28e81f161acd96f87 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e76b19894564aa28e81f161acd96f87" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 43645 } }
I20260926 15:12:15.088326 27425 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2e76b19894564aa28e81f161acd96f87 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e76b19894564aa28e81f161acd96f87, State: Initialized, Role: FOLLOWER
I20260926 15:12:15.088963 27425 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2e76b19894564aa28e81f161acd96f87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e76b19894564aa28e81f161acd96f87" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 43645 } }
I20260926 15:12:15.089252 27425 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2e76b19894564aa28e81f161acd96f87 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 15:12:15.089390 27425 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2e76b19894564aa28e81f161acd96f87 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 15:12:15.089564 27425 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2e76b19894564aa28e81f161acd96f87 [term 2 FOLLOWER]: Advancing to term 3
I20260926 15:12:15.093375 27425 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2e76b19894564aa28e81f161acd96f87 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e76b19894564aa28e81f161acd96f87" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 43645 } }
I20260926 15:12:15.093887 27425 leader_election.cc:302] T 00000000000000000000000000000000 P 2e76b19894564aa28e81f161acd96f87 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2e76b19894564aa28e81f161acd96f87; no voters:
I20260926 15:12:15.094602 27425 leader_election.cc:288] T 00000000000000000000000000000000 P 2e76b19894564aa28e81f161acd96f87 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260926 15:12:15.094746 27429 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2e76b19894564aa28e81f161acd96f87 [term 3 FOLLOWER]: Leader election won for term 3
I20260926 15:12:15.095834 27429 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2e76b19894564aa28e81f161acd96f87 [term 3 LEADER]: Becoming Leader. State: Replica: 2e76b19894564aa28e81f161acd96f87, State: Running, Role: LEADER
I20260926 15:12:15.096378 27429 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2e76b19894564aa28e81f161acd96f87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e76b19894564aa28e81f161acd96f87" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 43645 } }
I20260926 15:12:15.096819 27425 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2e76b19894564aa28e81f161acd96f87 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:12:15.098951 27431 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e76b19894564aa28e81f161acd96f87 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2e76b19894564aa28e81f161acd96f87. Latest consensus state: current_term: 3 leader_uuid: "2e76b19894564aa28e81f161acd96f87" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e76b19894564aa28e81f161acd96f87" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 43645 } } }
I20260926 15:12:15.099323 27431 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e76b19894564aa28e81f161acd96f87 [sys.catalog]: This master's current role is: LEADER
I20260926 15:12:15.099026 27430 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e76b19894564aa28e81f161acd96f87 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "2e76b19894564aa28e81f161acd96f87" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e76b19894564aa28e81f161acd96f87" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 43645 } } }
I20260926 15:12:15.099745 27430 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e76b19894564aa28e81f161acd96f87 [sys.catalog]: This master's current role is: LEADER
I20260926 15:12:15.109808 27435 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:12:15.119339 27435 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 15:12:15.120846 27435 catalog_manager.cc:1259] Loaded cluster ID: 5b907a4a881e41c48bab7c4645ef1315
I20260926 15:12:15.120947 27435 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260926 15:12:15.123291 27435 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7efd663a59ad google::LogMessage::Fail() at ??:0
@ 0x7efd663a888a google::LogMessage::SendToLog() at ??:0
@ 0x7efd663a54de google::LogMessage::Flush() at ??:0
@ 0x7efd663a6a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7efd7bdd8512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7efd7bdd5976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7efd7be5d610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7efd7be5d439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7efd68ff1345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7efd68ff491c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7efd68ff4799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7efd68fcfeaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7efd6c2a96db start_thread at ??:0
@ 0x7efd6523f61f clone at ??:0
W20260926 15:12:15.440151 27427 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:12:15.440402 27427 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:12:15.440496 27427 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:12:15.447556 27427 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:12:15.447654 27427 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:12:15.447769 27427 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.34.193
I20260926 15:12:15.457286 27427 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.34.212:36007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.193.keytab
--principal=kudu/127.12.34.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.34.193:35363
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.34.193
--webserver_port=33943
--tserver_master_addrs=127.12.34.254:43645
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.34.193
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:12:15.458367 27427 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:12:15.459635 27427 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:12:15.478390 27467 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:12:15.478446 27466 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:12:15.478477 27469 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:12:15.482561 27427 server_base.cc:1034] running on GCE node
I20260926 15:12:15.483171 27427 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:12:15.484073 27427 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:12:15.485275 27427 hybrid_clock.cc:648] HybridClock initialized: now 1790435535485233 us; error 55 us; skew 500 ppm
I20260926 15:12:15.485589 27427 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:12:15 dist-test-slave-865v krb5kdc[26274](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435535, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:12:15.488570 27427 init.cc:377] Logged in from keytab as kudu/127.12.34.193@KRBTEST.COM (short username kudu)
I20260926 15:12:15.490074 27427 webserver.cc:460] Webserver started at http://127.12.34.193:33943/ using document root <none> and password file <none>
I20260926 15:12:15.490800 27427 fs_manager.cc:359] Metadata directory not provided
I20260926 15:12:15.490947 27427 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:12:15.495954 27427 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 15:12:15.498767 27476 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:12:15.500073 27427 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 15:12:15.500237 27427 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/ts-0/data,/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/ts-0/wal
uuid: "4067cd15627b4f5ba7f5eb07551e7fab"
format_stamp: "Formatted at 2026-09-26 15:12:09 on dist-test-slave-865v"
server_key: "26f9593fa47ac2148c4a20256e4c7f86"
server_key_iv: "2c7606219edfd0d3e9eea39aa26ad591"
server_key_version: "encryptionkey@0"
I20260926 15:12:15.501256 27427 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:12:15.514261 27427 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:12:15.518195 27427 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:12:15.518471 27427 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:12:15.519410 27427 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:12:15.521124 27483 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260926 15:12:15.526440 27427 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 15:12:15.526597 27427 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20260926 15:12:15.526749 27427 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 15:12:15.529904 27427 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 15:12:15.530035 27427 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.003s sys 0.000s
I20260926 15:12:15.530453 27483 tablet_bootstrap.cc:492] T d164eccab1a9407d8cd8b6acc5aad66b P 4067cd15627b4f5ba7f5eb07551e7fab: Bootstrap starting.
I20260926 15:12:15.538477 27483 log.cc:824] T d164eccab1a9407d8cd8b6acc5aad66b P 4067cd15627b4f5ba7f5eb07551e7fab: Log is configured to *not* fsync() on all Append() calls
I20260926 15:12:15.555588 27483 tablet_bootstrap.cc:492] T d164eccab1a9407d8cd8b6acc5aad66b P 4067cd15627b4f5ba7f5eb07551e7fab: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 15:12:15.556403 27483 tablet_bootstrap.cc:492] T d164eccab1a9407d8cd8b6acc5aad66b P 4067cd15627b4f5ba7f5eb07551e7fab: Bootstrap complete.
I20260926 15:12:15.557199 27483 ts_tablet_manager.cc:1397] T d164eccab1a9407d8cd8b6acc5aad66b P 4067cd15627b4f5ba7f5eb07551e7fab: Time spent bootstrapping tablet: real 0.027s user 0.009s sys 0.016s
I20260926 15:12:15.565395 27483 raft_consensus.cc:348] T d164eccab1a9407d8cd8b6acc5aad66b P 4067cd15627b4f5ba7f5eb07551e7fab [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4067cd15627b4f5ba7f5eb07551e7fab" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 35363 } } peers { permanent_uuid: "b440b82da2b24d09ac47aceedd225013" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 35357 } } peers { permanent_uuid: "cbe51bf32d864c85a4189dc5531617b1" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 37165 } }
I20260926 15:12:15.566769 27483 raft_consensus.cc:729] T d164eccab1a9407d8cd8b6acc5aad66b P 4067cd15627b4f5ba7f5eb07551e7fab [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4067cd15627b4f5ba7f5eb07551e7fab, State: Initialized, Role: FOLLOWER
I20260926 15:12:15.567605 27483 consensus_queue.cc:260] T d164eccab1a9407d8cd8b6acc5aad66b P 4067cd15627b4f5ba7f5eb07551e7fab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4067cd15627b4f5ba7f5eb07551e7fab" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 35363 } } peers { permanent_uuid: "b440b82da2b24d09ac47aceedd225013" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 35357 } } peers { permanent_uuid: "cbe51bf32d864c85a4189dc5531617b1" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 37165 } }
I20260926 15:12:15.569151 27483 ts_tablet_manager.cc:1428] T d164eccab1a9407d8cd8b6acc5aad66b P 4067cd15627b4f5ba7f5eb07551e7fab: Time spent starting tablet: real 0.012s user 0.005s sys 0.006s
I20260926 15:12:15.575305 27427 rpc_server.cc:307] RPC server started. Bound to: 127.12.34.193:35363
I20260926 15:12:15.575388 27591 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.34.193:35363 every 8 connection(s)
I20260926 15:12:15.576608 27427 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/ts-0/data/info.pb
I20260926 15:12:15.580996 27594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:15.579270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:43645 (local address 127.12.34.193:49645)
0926 15:12:15.579994 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0926 15:12:15.580213 (+ 219us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.254:43645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":289,"threads_started":1}
W20260926 15:12:15.581620 27592 heartbeater.cc:644] Failed to heartbeat to 127.12.34.254:43645 (0 consecutive failures): Network error: Failed to ping master at 127.12.34.254:43645: Client connection negotiation failed: client connection to 127.12.34.254:43645: connect: Connection refused (error 111)
I20260926 15:12:15.581907 12427 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8uoJGv/build/asan/bin/kudu as pid 27427
I20260926 15:12:15.582406 27594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:15.582090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:43645 (local address 127.12.34.193:36475)
0926 15:12:15.582234 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0926 15:12:15.582310 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.254:43645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260926 15:12:15.582589 12427 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8uoJGv/build/asan/bin/kudu
/tmp/dist-test-task8uoJGv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.34.194:37165
--local_ip_for_outbound_sockets=127.12.34.194
--tserver_master_addrs=127.12.34.254:43645
--webserver_port=38851
--webserver_interface=127.12.34.194
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.194.keytab
--principal=kudu/127.12.34.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.34.212:36007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 15:12:15.583359 27594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:15.582889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:43645 (local address 127.12.34.193:40945)
0926 15:12:15.583072 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0926 15:12:15.583183 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.254:43645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260926 15:12:15.583630 27592 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 15:12:15.902515 27595 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:12:15.902796 27595 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:12:15.902964 27595 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:12:15.910581 27595 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:12:15.910696 27595 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:12:15.910876 27595 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.34.194
I20260926 15:12:15.920346 27595 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.34.212:36007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.194.keytab
--principal=kudu/127.12.34.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.34.194:37165
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.34.194
--webserver_port=38851
--tserver_master_addrs=127.12.34.254:43645
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.34.194
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:12:15.921667 27595 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:12:15.922936 27595 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:12:15.935806 27600 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:12:15.935859 27603 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:12:15.935897 27601 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:12:15.936654 27595 server_base.cc:1034] running on GCE node
I20260926 15:12:15.937229 27595 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:12:15.938138 27595 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:12:15.939360 27595 hybrid_clock.cc:648] HybridClock initialized: now 1790435535939223 us; error 121 us; skew 500 ppm
I20260926 15:12:15.939698 27595 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:12:15 dist-test-slave-865v krb5kdc[26274](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435535, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:12:15.943025 27595 init.cc:377] Logged in from keytab as kudu/127.12.34.194@KRBTEST.COM (short username kudu)
I20260926 15:12:15.944485 27595 webserver.cc:460] Webserver started at http://127.12.34.194:38851/ using document root <none> and password file <none>
I20260926 15:12:15.945218 27595 fs_manager.cc:359] Metadata directory not provided
I20260926 15:12:15.945345 27595 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:12:15.950603 27595 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 15:12:15.953293 27610 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:12:15.954624 27595 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 15:12:15.954811 27595 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/ts-1/data,/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/ts-1/wal
uuid: "cbe51bf32d864c85a4189dc5531617b1"
format_stamp: "Formatted at 2026-09-26 15:12:10 on dist-test-slave-865v"
server_key: "b57409c04ecb3bf07d4f7f77e12c6e23"
server_key_iv: "807e84f89a714d99727fafcd477a3854"
server_key_version: "encryptionkey@0"
I20260926 15:12:15.955806 27595 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:12:15.985983 27595 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:12:15.989969 27595 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:12:15.990247 27595 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:12:15.991169 27595 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:12:15.992455 27595 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:12:15.992576 27595 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:12:15.992698 27595 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:12:15.992787 27595 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:12:16.026080 27595 rpc_server.cc:307] RPC server started. Bound to: 127.12.34.194:37165
I20260926 15:12:16.026161 27722 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.34.194:37165 every 8 connection(s)
I20260926 15:12:16.027458 27595 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/ts-1/data/info.pb
I20260926 15:12:16.031862 27725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:16.030171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:43645 (local address 127.12.34.194:58009)
0926 15:12:16.030827 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0926 15:12:16.031052 (+ 225us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.254:43645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":261,"threads_started":1}
W20260926 15:12:16.032642 27723 heartbeater.cc:644] Failed to heartbeat to 127.12.34.254:43645 (0 consecutive failures): Network error: Failed to ping master at 127.12.34.254:43645: Client connection negotiation failed: client connection to 127.12.34.254:43645: connect: Connection refused (error 111)
I20260926 15:12:16.033443 27725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:16.033160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:43645 (local address 127.12.34.194:57779)
0926 15:12:16.033287 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0926 15:12:16.033348 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.254:43645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260926 15:12:16.034265 27725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:16.033984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:43645 (local address 127.12.34.194:33129)
0926 15:12:16.034099 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0926 15:12:16.034164 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.254:43645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260926 15:12:16.034498 27723 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 15:12:16.035038 12427 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8uoJGv/build/asan/bin/kudu as pid 27595
I20260926 15:12:16.035612 12427 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8uoJGv/build/asan/bin/kudu
/tmp/dist-test-task8uoJGv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.34.195:35357
--local_ip_for_outbound_sockets=127.12.34.195
--tserver_master_addrs=127.12.34.254:43645
--webserver_port=35437
--webserver_interface=127.12.34.195
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.195.keytab
--principal=kudu/127.12.34.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.34.212:36007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:12:16.349018 27726 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:12:16.349287 27726 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:12:16.349378 27726 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:12:16.356474 27726 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:12:16.356567 27726 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:12:16.356674 27726 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.34.195
I20260926 15:12:16.366137 27726 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.34.212:36007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.195.keytab
--principal=kudu/127.12.34.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.34.195:35357
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.34.195
--webserver_port=35437
--tserver_master_addrs=127.12.34.254:43645
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.34.195
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:12:16.367287 27726 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:12:16.368465 27726 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:12:16.380869 27731 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:12:16.380999 27732 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:12:16.380973 27734 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:12:16.382164 27726 server_base.cc:1034] running on GCE node
I20260926 15:12:16.382704 27726 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:12:16.383599 27726 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:12:16.384821 27726 hybrid_clock.cc:648] HybridClock initialized: now 1790435536384781 us; error 24 us; skew 500 ppm
I20260926 15:12:16.385208 27726 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:12:16 dist-test-slave-865v krb5kdc[26274](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435536, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:12:16.388346 27726 init.cc:377] Logged in from keytab as kudu/127.12.34.195@KRBTEST.COM (short username kudu)
I20260926 15:12:16.389842 27726 webserver.cc:460] Webserver started at http://127.12.34.195:35437/ using document root <none> and password file <none>
I20260926 15:12:16.390488 27726 fs_manager.cc:359] Metadata directory not provided
I20260926 15:12:16.390630 27726 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:12:16.395656 27726 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 15:12:16.398411 27741 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:12:16.399732 27726 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 15:12:16.399936 27726 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/ts-2/data,/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/ts-2/wal
uuid: "b440b82da2b24d09ac47aceedd225013"
format_stamp: "Formatted at 2026-09-26 15:12:10 on dist-test-slave-865v"
server_key: "12aec6bcbdef59bc94e5bbd9e3b0a73b"
server_key_iv: "5a9e03d0524afb3807e1879b21bc934c"
server_key_version: "encryptionkey@0"
I20260926 15:12:16.400866 27726 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:12:16.416189 27726 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:12:16.420176 27726 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:12:16.420447 27726 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:12:16.421424 27726 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:12:16.422650 27726 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:12:16.422777 27726 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:12:16.422953 27726 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:12:16.423049 27726 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:12:16.458561 27726 rpc_server.cc:307] RPC server started. Bound to: 127.12.34.195:35357
I20260926 15:12:16.458653 27853 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.34.195:35357 every 8 connection(s)
I20260926 15:12:16.460183 27726 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790435318278681-12427-0/minicluster-data/ts-2/data/info.pb
I20260926 15:12:16.465036 27856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:16.463265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:43645 (local address 127.12.34.195:46675)
0926 15:12:16.464007 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0926 15:12:16.464215 (+ 208us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.254:43645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":288,"threads_started":1}
W20260926 15:12:16.465916 27854 heartbeater.cc:644] Failed to heartbeat to 127.12.34.254:43645 (0 consecutive failures): Network error: Failed to ping master at 127.12.34.254:43645: Client connection negotiation failed: client connection to 127.12.34.254:43645: connect: Connection refused (error 111)
I20260926 15:12:16.466922 27856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:16.466594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:43645 (local address 127.12.34.195:47435)
0926 15:12:16.466743 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0926 15:12:16.466809 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.254:43645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260926 15:12:16.467562 12427 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8uoJGv/build/asan/bin/kudu as pid 27726
I20260926 15:12:16.468215 27856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:16.467823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:43645 (local address 127.12.34.195:46677)
0926 15:12:16.467959 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0926 15:12:16.468051 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.254:43645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260926 15:12:16.468530 27854 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 15:12:16.584794 27858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:16.584100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:43645 (local address 127.12.34.193:45103)
0926 15:12:16.584481 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0926 15:12:16.584592 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.254:43645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":182,"threads_started":1}
I20260926 15:12:16.981731 27859 raft_consensus.cc:482] T d164eccab1a9407d8cd8b6acc5aad66b P 4067cd15627b4f5ba7f5eb07551e7fab [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 15:12:16.982084 27859 raft_consensus.cc:504] T d164eccab1a9407d8cd8b6acc5aad66b P 4067cd15627b4f5ba7f5eb07551e7fab [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4067cd15627b4f5ba7f5eb07551e7fab" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 35363 } } peers { permanent_uuid: "b440b82da2b24d09ac47aceedd225013" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 35357 } } peers { permanent_uuid: "cbe51bf32d864c85a4189dc5531617b1" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 37165 } }
I20260926 15:12:16.983804 27859 leader_election.cc:288] T d164eccab1a9407d8cd8b6acc5aad66b P 4067cd15627b4f5ba7f5eb07551e7fab [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b440b82da2b24d09ac47aceedd225013 (127.12.34.195:35357), cbe51bf32d864c85a4189dc5531617b1 (127.12.34.194:37165)
Sep 26 15:12:16 dist-test-slave-865v krb5kdc[26274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435535, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.193@KRBTEST.COM for kudu/127.12.34.195@KRBTEST.COM
Sep 26 15:12:16 dist-test-slave-865v krb5kdc[26274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435535, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.193@KRBTEST.COM for kudu/127.12.34.194@KRBTEST.COM
I20260926 15:12:17.003458 27858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:16.984079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.195:35357 (local address 127.12.34.193:36303)
0926 15:12:16.984264 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0926 15:12:16.984301 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 15:12:16.984855 (+ 554us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:12:16.987378 (+ 2523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:12:16.987391 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:12:16.988105 (+ 714us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:12:16.988766 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:12:16.988785 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:12:16.990362 (+ 1577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:12:16.990366 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:12:16.991709 (+ 1343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:12:16.991720 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:12:16.992670 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:12:16.992676 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:12:16.992922 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:12:16.993727 (+ 805us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:12:16.993750 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:12:16.996086 (+ 2336us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:12:16.999576 (+ 3490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:12:16.999586 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:12:16.999601 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:12:17.000195 (+ 594us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:12:17.000719 (+ 524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:12:17.000722 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:12:17.000724 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:12:17.000889 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:12:17.001860 (+ 971us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:12:17.001867 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:12:17.002237 (+ 370us) client_negotiation.cc:769] Sending connection context
0926 15:12:17.002566 (+ 329us) client_negotiation.cc:240] Negotiation successful
0926 15:12:17.003006 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54,"mutex_wait_us":101}
I20260926 15:12:17.003641 27860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:16.984263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.193:36303 (local address 127.12.34.195:35357)
0926 15:12:16.984788 (+ 525us) server_negotiation.cc:206] Beginning negotiation
0926 15:12:16.984804 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 15:12:16.984863 (+ 59us) server_negotiation.cc:407] Connection header received
0926 15:12:16.986586 (+ 1723us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:12:16.986595 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:12:16.986956 (+ 361us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:12:16.987264 (+ 308us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:12:16.988955 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:12:16.990158 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:12:16.991862 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:12:16.992459 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:12:16.992683 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:12:16.996383 (+ 3700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:12:16.996396 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:12:16.996416 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:12:16.996460 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:12:16.999414 (+ 2954us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:12:17.000359 (+ 945us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:12:17.000368 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:12:17.000376 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:12:17.000606 (+ 230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:12:17.001039 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:12:17.001043 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:12:17.001045 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:12:17.001723 (+ 678us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:12:17.001880 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0926 15:12:17.002807 (+ 927us) server_negotiation.cc:299] Negotiation successful
0926 15:12:17.003225 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":273,"threads_started":1}
I20260926 15:12:17.003824 27862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:16.985018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.193:56131 (local address 127.12.34.194:37165)
0926 15:12:16.985585 (+ 567us) server_negotiation.cc:206] Beginning negotiation
0926 15:12:16.985593 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 15:12:16.985620 (+ 27us) server_negotiation.cc:407] Connection header received
0926 15:12:16.986978 (+ 1358us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:12:16.986986 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:12:16.987281 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:12:16.987567 (+ 286us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:12:16.988963 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:12:16.990164 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:12:16.991860 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:12:16.992462 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:12:16.992674 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:12:16.996795 (+ 4121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:12:16.996802 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:12:16.996818 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:12:16.996856 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:12:17.000195 (+ 3339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:12:17.001036 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:12:17.001044 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:12:17.001058 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:12:17.001282 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:12:17.001683 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:12:17.001686 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:12:17.001688 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:12:17.002281 (+ 593us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:12:17.002474 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0926 15:12:17.003000 (+ 526us) server_negotiation.cc:299] Negotiation successful
0926 15:12:17.003377 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":425,"thread_start_us":327,"threads_started":1}
I20260926 15:12:17.003692 27861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:16.984481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.194:37165 (local address 127.12.34.193:56131)
0926 15:12:16.985173 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0926 15:12:16.985191 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 15:12:16.985307 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:12:16.987529 (+ 2222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:12:16.987533 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:12:16.988177 (+ 644us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:12:16.988768 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:12:16.988787 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:12:16.990336 (+ 1549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:12:16.990350 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:12:16.991712 (+ 1362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:12:16.991724 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:12:16.992981 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:12:16.992989 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:12:16.993185 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:12:16.994105 (+ 920us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:12:16.994122 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:12:16.996487 (+ 2365us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:12:17.000368 (+ 3881us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:12:17.000373 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:12:17.000377 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:12:17.000881 (+ 504us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:12:17.001397 (+ 516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:12:17.001402 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:12:17.001405 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:12:17.001571 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:12:17.002429 (+ 858us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:12:17.002432 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:12:17.002606 (+ 174us) client_negotiation.cc:769] Sending connection context
0926 15:12:17.002764 (+ 158us) client_negotiation.cc:240] Negotiation successful
0926 15:12:17.003175 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"mutex_wait_us":197,"thread_start_us":685,"threads_started":1}
W20260926 15:12:17.007550 27477 leader_election.cc:341] T d164eccab1a9407d8cd8b6acc5aad66b P 4067cd15627b4f5ba7f5eb07551e7fab [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cbe51bf32d864c85a4189dc5531617b1 (127.12.34.194:37165): Not found: Tablet not found: d164eccab1a9407d8cd8b6acc5aad66b
W20260926 15:12:17.007853 27477 leader_election.cc:341] T d164eccab1a9407d8cd8b6acc5aad66b P 4067cd15627b4f5ba7f5eb07551e7fab [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b440b82da2b24d09ac47aceedd225013 (127.12.34.195:35357): Not found: Tablet not found: d164eccab1a9407d8cd8b6acc5aad66b
I20260926 15:12:17.008008 27477 leader_election.cc:302] T d164eccab1a9407d8cd8b6acc5aad66b P 4067cd15627b4f5ba7f5eb07551e7fab [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4067cd15627b4f5ba7f5eb07551e7fab; no voters: b440b82da2b24d09ac47aceedd225013, cbe51bf32d864c85a4189dc5531617b1
I20260926 15:12:17.008428 27859 raft_consensus.cc:2726] T d164eccab1a9407d8cd8b6acc5aad66b P 4067cd15627b4f5ba7f5eb07551e7fab [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 15:12:17.035925 27863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:17.035024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:43645 (local address 127.12.34.194:38151)
0926 15:12:17.035496 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0926 15:12:17.035663 (+ 167us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.254:43645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":228,"threads_started":1}
I20260926 15:12:17.469978 27864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:17.469040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:43645 (local address 127.12.34.195:47327)
0926 15:12:17.469615 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0926 15:12:17.469746 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.254:43645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":332,"threads_started":1}
I20260926 15:12:17.586364 27865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:17.585659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:43645 (local address 127.12.34.193:40509)
0926 15:12:17.586114 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0926 15:12:17.586210 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.254:43645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":229,"threads_started":1}
I20260926 15:12:18.037693 27866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:18.036992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:43645 (local address 127.12.34.194:32875)
0926 15:12:18.037451 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0926 15:12:18.037543 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.254:43645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":253,"threads_started":1}
I20260926 15:12:18.471740 27867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:18.470811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:43645 (local address 127.12.34.195:39565)
0926 15:12:18.471386 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0926 15:12:18.471505 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.254:43645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":325,"threads_started":1}
I20260926 15:12:18.587883 27868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:18.587213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:43645 (local address 127.12.34.193:40675)
0926 15:12:18.587639 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0926 15:12:18.587716 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.254:43645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":212,"threads_started":1}
I20260926 15:12:18.812623 27869 raft_consensus.cc:482] T d164eccab1a9407d8cd8b6acc5aad66b P 4067cd15627b4f5ba7f5eb07551e7fab [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 15:12:18.812856 27869 raft_consensus.cc:504] T d164eccab1a9407d8cd8b6acc5aad66b P 4067cd15627b4f5ba7f5eb07551e7fab [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4067cd15627b4f5ba7f5eb07551e7fab" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 35363 } } peers { permanent_uuid: "b440b82da2b24d09ac47aceedd225013" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 35357 } } peers { permanent_uuid: "cbe51bf32d864c85a4189dc5531617b1" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 37165 } }
I20260926 15:12:18.813526 27869 leader_election.cc:288] T d164eccab1a9407d8cd8b6acc5aad66b P 4067cd15627b4f5ba7f5eb07551e7fab [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b440b82da2b24d09ac47aceedd225013 (127.12.34.195:35357), cbe51bf32d864c85a4189dc5531617b1 (127.12.34.194:37165)
W20260926 15:12:18.814761 27477 leader_election.cc:341] T d164eccab1a9407d8cd8b6acc5aad66b P 4067cd15627b4f5ba7f5eb07551e7fab [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b440b82da2b24d09ac47aceedd225013 (127.12.34.195:35357): Not found: Tablet not found: d164eccab1a9407d8cd8b6acc5aad66b
W20260926 15:12:18.814970 27477 leader_election.cc:341] T d164eccab1a9407d8cd8b6acc5aad66b P 4067cd15627b4f5ba7f5eb07551e7fab [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cbe51bf32d864c85a4189dc5531617b1 (127.12.34.194:37165): Not found: Tablet not found: d164eccab1a9407d8cd8b6acc5aad66b
I20260926 15:12:18.815061 27477 leader_election.cc:302] T d164eccab1a9407d8cd8b6acc5aad66b P 4067cd15627b4f5ba7f5eb07551e7fab [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4067cd15627b4f5ba7f5eb07551e7fab; no voters: b440b82da2b24d09ac47aceedd225013, cbe51bf32d864c85a4189dc5531617b1
I20260926 15:12:18.815272 27869 raft_consensus.cc:2726] T d164eccab1a9407d8cd8b6acc5aad66b P 4067cd15627b4f5ba7f5eb07551e7fab [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 15:12:19.039456 27870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:19.038661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:43645 (local address 127.12.34.194:47957)
0926 15:12:19.039204 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0926 15:12:19.039287 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.254:43645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":342,"threads_started":1}
I20260926 15:12:19.473325 27871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:19.472583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:43645 (local address 127.12.34.195:51849)
0926 15:12:19.473031 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0926 15:12:19.473169 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.254:43645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":212,"threads_started":1}
I20260926 15:12:19.589458 27872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:19.588710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:43645 (local address 127.12.34.193:43379)
0926 15:12:19.589214 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0926 15:12:19.589295 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.254:43645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":238,"threads_started":1}
I20260926 15:12:20.041127 27873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:20.040390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:43645 (local address 127.12.34.194:47731)
0926 15:12:20.040830 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0926 15:12:20.040909 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.254:43645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":221,"threads_started":1}
I20260926 15:12:20.426319 27874 raft_consensus.cc:482] T d164eccab1a9407d8cd8b6acc5aad66b P 4067cd15627b4f5ba7f5eb07551e7fab [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 15:12:20.426550 27874 raft_consensus.cc:504] T d164eccab1a9407d8cd8b6acc5aad66b P 4067cd15627b4f5ba7f5eb07551e7fab [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4067cd15627b4f5ba7f5eb07551e7fab" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 35363 } } peers { permanent_uuid: "b440b82da2b24d09ac47aceedd225013" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 35357 } } peers { permanent_uuid: "cbe51bf32d864c85a4189dc5531617b1" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 37165 } }
I20260926 15:12:20.427225 27874 leader_election.cc:288] T d164eccab1a9407d8cd8b6acc5aad66b P 4067cd15627b4f5ba7f5eb07551e7fab [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b440b82da2b24d09ac47aceedd225013 (127.12.34.195:35357), cbe51bf32d864c85a4189dc5531617b1 (127.12.34.194:37165)
W20260926 15:12:20.428164 27477 leader_election.cc:341] T d164eccab1a9407d8cd8b6acc5aad66b P 4067cd15627b4f5ba7f5eb07551e7fab [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b440b82da2b24d09ac47aceedd225013 (127.12.34.195:35357): Not found: Tablet not found: d164eccab1a9407d8cd8b6acc5aad66b
W20260926 15:12:20.428380 27477 leader_election.cc:341] T d164eccab1a9407d8cd8b6acc5aad66b P 4067cd15627b4f5ba7f5eb07551e7fab [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cbe51bf32d864c85a4189dc5531617b1 (127.12.34.194:37165): Not found: Tablet not found: d164eccab1a9407d8cd8b6acc5aad66b
I20260926 15:12:20.428481 27477 leader_election.cc:302] T d164eccab1a9407d8cd8b6acc5aad66b P 4067cd15627b4f5ba7f5eb07551e7fab [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4067cd15627b4f5ba7f5eb07551e7fab; no voters: b440b82da2b24d09ac47aceedd225013, cbe51bf32d864c85a4189dc5531617b1
I20260926 15:12:20.428789 27874 raft_consensus.cc:2726] T d164eccab1a9407d8cd8b6acc5aad66b P 4067cd15627b4f5ba7f5eb07551e7fab [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 15:12:20.474862 27875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:20.474197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:43645 (local address 127.12.34.195:45123)
0926 15:12:20.474613 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0926 15:12:20.474689 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.254:43645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":195,"threads_started":1}
I20260926 15:12:20.591120 27876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:20.590381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:43645 (local address 127.12.34.193:34819)
0926 15:12:20.590848 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0926 15:12:20.590942 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.254:43645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":212,"threads_started":1}
I20260926 15:12:21.042780 27877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:21.042096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:43645 (local address 127.12.34.194:35907)
0926 15:12:21.042536 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0926 15:12:21.042618 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.254:43645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":227,"threads_started":1}
I20260926 15:12:21.476478 27878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:21.475762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:43645 (local address 127.12.34.195:44775)
0926 15:12:21.476212 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0926 15:12:21.476305 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.254:43645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":257,"threads_started":1}
I20260926 15:12:21.592954 27879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:21.592097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:43645 (local address 127.12.34.193:57439)
0926 15:12:21.592589 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0926 15:12:21.592702 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.254:43645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":215,"threads_started":1}
I20260926 15:12:22.044447 27880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:22.043689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:43645 (local address 127.12.34.194:35919)
0926 15:12:22.044192 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0926 15:12:22.044271 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.254:43645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":272,"threads_started":1}
I20260926 15:12:22.478099 27881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:22.477402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:43645 (local address 127.12.34.195:45229)
0926 15:12:22.477847 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0926 15:12:22.477931 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.254:43645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":224,"threads_started":1}
I20260926 15:12:22.594544 27882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:22.593893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:43645 (local address 127.12.34.193:33359)
0926 15:12:22.594278 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0926 15:12:22.594354 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.254:43645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":202,"threads_started":1}
I20260926 15:12:23.046303 27883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:23.045489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:43645 (local address 127.12.34.194:45931)
0926 15:12:23.046018 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0926 15:12:23.046108 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.254:43645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":250,"threads_started":1}
I20260926 15:12:23.479882 27884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:23.479001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:43645 (local address 127.12.34.195:41533)
0926 15:12:23.479600 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0926 15:12:23.479684 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.254:43645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":328,"threads_started":1}
I20260926 15:12:23.596623 27885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:23.595501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:43645 (local address 127.12.34.193:59649)
0926 15:12:23.596340 (+ 839us) negotiation.cc:107] Waiting for socket to connect
0926 15:12:23.596451 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.254:43645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":719,"thread_start_us":468,"threads_started":1}
I20260926 15:12:23.651585 27886 raft_consensus.cc:482] T d164eccab1a9407d8cd8b6acc5aad66b P 4067cd15627b4f5ba7f5eb07551e7fab [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 15:12:23.651837 27886 raft_consensus.cc:504] T d164eccab1a9407d8cd8b6acc5aad66b P 4067cd15627b4f5ba7f5eb07551e7fab [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4067cd15627b4f5ba7f5eb07551e7fab" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 35363 } } peers { permanent_uuid: "b440b82da2b24d09ac47aceedd225013" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 35357 } } peers { permanent_uuid: "cbe51bf32d864c85a4189dc5531617b1" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 37165 } }
I20260926 15:12:23.652527 27886 leader_election.cc:288] T d164eccab1a9407d8cd8b6acc5aad66b P 4067cd15627b4f5ba7f5eb07551e7fab [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b440b82da2b24d09ac47aceedd225013 (127.12.34.195:35357), cbe51bf32d864c85a4189dc5531617b1 (127.12.34.194:37165)
W20260926 15:12:23.653558 27477 leader_election.cc:341] T d164eccab1a9407d8cd8b6acc5aad66b P 4067cd15627b4f5ba7f5eb07551e7fab [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b440b82da2b24d09ac47aceedd225013 (127.12.34.195:35357): Not found: Tablet not found: d164eccab1a9407d8cd8b6acc5aad66b
W20260926 15:12:23.653748 27477 leader_election.cc:341] T d164eccab1a9407d8cd8b6acc5aad66b P 4067cd15627b4f5ba7f5eb07551e7fab [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cbe51bf32d864c85a4189dc5531617b1 (127.12.34.194:37165): Not found: Tablet not found: d164eccab1a9407d8cd8b6acc5aad66b
I20260926 15:12:23.653820 27477 leader_election.cc:302] T d164eccab1a9407d8cd8b6acc5aad66b P 4067cd15627b4f5ba7f5eb07551e7fab [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4067cd15627b4f5ba7f5eb07551e7fab; no voters: b440b82da2b24d09ac47aceedd225013, cbe51bf32d864c85a4189dc5531617b1
I20260926 15:12:23.654039 27886 raft_consensus.cc:2726] T d164eccab1a9407d8cd8b6acc5aad66b P 4067cd15627b4f5ba7f5eb07551e7fab [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 15:12:24.048054 27887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:24.047326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:43645 (local address 127.12.34.194:40705)
0926 15:12:24.047801 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0926 15:12:24.047882 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.254:43645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":250,"threads_started":1}
I20260926 15:12:24.481703 27888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:24.480766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:43645 (local address 127.12.34.195:38197)
0926 15:12:24.481414 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0926 15:12:24.481523 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.254:43645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":307,"threads_started":1}
I20260926 15:12:24.598273 27889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:24.597576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:43645 (local address 127.12.34.193:37693)
0926 15:12:24.598029 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0926 15:12:24.598110 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.254:43645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":224,"threads_started":1}
I20260926 15:12:25.049639 27890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:25.048901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:43645 (local address 127.12.34.194:35293)
0926 15:12:25.049386 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0926 15:12:25.049470 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.254:43645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":273,"threads_started":1}
I20260926 15:12:25.483431 27891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:25.482697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:43645 (local address 127.12.34.195:46493)
0926 15:12:25.483181 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0926 15:12:25.483265 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.254:43645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":264,"threads_started":1}
I20260926 15:12:25.599864 27892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:25.599184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:43645 (local address 127.12.34.193:41157)
0926 15:12:25.599626 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0926 15:12:25.599704 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.254:43645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":235,"threads_started":1}
I20260926 15:12:26.051291 27893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:26.050553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:43645 (local address 127.12.34.194:53971)
0926 15:12:26.051030 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0926 15:12:26.051117 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.254:43645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":251,"threads_started":1}
I20260926 15:12:26.485249 27894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:26.484460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:43645 (local address 127.12.34.195:42347)
0926 15:12:26.484910 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0926 15:12:26.484997 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.254:43645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":245,"threads_started":1}
I20260926 15:12:26.601636 27895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:26.600862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:43645 (local address 127.12.34.193:56025)
0926 15:12:26.601349 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0926 15:12:26.601435 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.254:43645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":301,"threads_started":1}
I20260926 15:12:27.053061 27896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:27.052320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:43645 (local address 127.12.34.194:38905)
0926 15:12:27.052775 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0926 15:12:27.052870 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.254:43645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":225,"threads_started":1}
I20260926 15:12:27.486752 27897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:27.486051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:43645 (local address 127.12.34.195:59093)
0926 15:12:27.486515 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0926 15:12:27.486597 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.254:43645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":231,"threads_started":1}
I20260926 15:12:27.603166 27898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:27.602506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:43645 (local address 127.12.34.193:50883)
0926 15:12:27.602921 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0926 15:12:27.603009 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.254:43645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":213,"threads_started":1}
I20260926 15:12:28.055373 27899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:28.054239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:43645 (local address 127.12.34.194:50699)
0926 15:12:28.054989 (+ 750us) negotiation.cc:107] Waiting for socket to connect
0926 15:12:28.055126 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.254:43645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":570,"thread_start_us":413,"threads_started":1}
I20260926 15:12:28.488385 27900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:28.487656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:43645 (local address 127.12.34.195:53439)
0926 15:12:28.488114 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0926 15:12:28.488196 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.254:43645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":257,"threads_started":1}
I20260926 15:12:28.605195 27901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:28.604181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:43645 (local address 127.12.34.193:49693)
0926 15:12:28.604776 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0926 15:12:28.604937 (+ 161us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.254:43645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":338,"threads_started":1}
I20260926 15:12:28.951140 27902 raft_consensus.cc:482] T d164eccab1a9407d8cd8b6acc5aad66b P 4067cd15627b4f5ba7f5eb07551e7fab [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 15:12:28.951327 27902 raft_consensus.cc:504] T d164eccab1a9407d8cd8b6acc5aad66b P 4067cd15627b4f5ba7f5eb07551e7fab [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4067cd15627b4f5ba7f5eb07551e7fab" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 35363 } } peers { permanent_uuid: "b440b82da2b24d09ac47aceedd225013" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 35357 } } peers { permanent_uuid: "cbe51bf32d864c85a4189dc5531617b1" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 37165 } }
I20260926 15:12:28.951934 27902 leader_election.cc:288] T d164eccab1a9407d8cd8b6acc5aad66b P 4067cd15627b4f5ba7f5eb07551e7fab [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b440b82da2b24d09ac47aceedd225013 (127.12.34.195:35357), cbe51bf32d864c85a4189dc5531617b1 (127.12.34.194:37165)
W20260926 15:12:28.952946 27477 leader_election.cc:341] T d164eccab1a9407d8cd8b6acc5aad66b P 4067cd15627b4f5ba7f5eb07551e7fab [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b440b82da2b24d09ac47aceedd225013 (127.12.34.195:35357): Not found: Tablet not found: d164eccab1a9407d8cd8b6acc5aad66b
W20260926 15:12:28.953208 27477 leader_election.cc:341] T d164eccab1a9407d8cd8b6acc5aad66b P 4067cd15627b4f5ba7f5eb07551e7fab [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cbe51bf32d864c85a4189dc5531617b1 (127.12.34.194:37165): Not found: Tablet not found: d164eccab1a9407d8cd8b6acc5aad66b
I20260926 15:12:28.953332 27477 leader_election.cc:302] T d164eccab1a9407d8cd8b6acc5aad66b P 4067cd15627b4f5ba7f5eb07551e7fab [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4067cd15627b4f5ba7f5eb07551e7fab; no voters: b440b82da2b24d09ac47aceedd225013, cbe51bf32d864c85a4189dc5531617b1
I20260926 15:12:28.953631 27902 raft_consensus.cc:2726] T d164eccab1a9407d8cd8b6acc5aad66b P 4067cd15627b4f5ba7f5eb07551e7fab [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 15:12:29.057348 27903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:29.056483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:43645 (local address 127.12.34.194:53283)
0926 15:12:29.057003 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0926 15:12:29.057165 (+ 162us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.254:43645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":245,"threads_started":1}
I20260926 15:12:29.490078 27904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:29.489370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:43645 (local address 127.12.34.195:32911)
0926 15:12:29.489808 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0926 15:12:29.489910 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.254:43645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":229,"threads_started":1}
I20260926 15:12:29.606618 27905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:29.605955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:43645 (local address 127.12.34.193:54965)
0926 15:12:29.606365 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0926 15:12:29.606449 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.254:43645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":220,"threads_started":1}
I20260926 15:12:30.059146 27906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:30.058333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:43645 (local address 127.12.34.194:52049)
0926 15:12:30.058869 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0926 15:12:30.058954 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.254:43645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":305,"threads_started":1}
I20260926 15:12:30.491834 27907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:30.490991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:43645 (local address 127.12.34.195:39719)
0926 15:12:30.491488 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0926 15:12:30.491590 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.254:43645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":241,"threads_started":1}
I20260926 15:12:30.608438 27908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:30.607663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:43645 (local address 127.12.34.193:48693)
0926 15:12:30.608143 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0926 15:12:30.608239 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.254:43645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":258,"threads_started":1}
I20260926 15:12:31.061282 27909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:31.060498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:43645 (local address 127.12.34.194:42915)
0926 15:12:31.060948 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0926 15:12:31.061055 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.254:43645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":220,"threads_started":1}
I20260926 15:12:31.469884 12427 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 27427
I20260926 15:12:31.493618 27910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:31.470936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33436 (local address 127.12.34.193:35363)
0926 15:12:31.471537 (+ 601us) server_negotiation.cc:206] Beginning negotiation
0926 15:12:31.471546 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 15:12:31.471755 (+ 209us) server_negotiation.cc:407] Connection header received
0926 15:12:31.471889 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:12:31.471898 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:12:31.472198 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:12:31.472404 (+ 206us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:12:31.474712 (+ 2308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:12:31.475660 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:12:31.476847 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:12:31.477428 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:12:31.477523 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:12:31.487879 (+ 10356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:12:31.487892 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:12:31.487914 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:12:31.487964 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:12:31.490772 (+ 2808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:12:31.491392 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:12:31.491399 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:12:31.491401 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:12:31.491535 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:12:31.491891 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:12:31.491895 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:12:31.491897 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:12:31.492259 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:12:31.492489 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0926 15:12:31.492918 (+ 429us) server_negotiation.cc:299] Negotiation successful
0926 15:12:31.493221 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":287,"threads_started":1}
I20260926 15:12:31.493954 27911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:31.492954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:43645 (local address 127.12.34.195:57439)
0926 15:12:31.493658 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0926 15:12:31.493755 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.254:43645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":578,"thread_start_us":459,"threads_started":1}
I20260926 15:12:31.494917 27587 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:12:31.586335 12427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8uoJGv/build/asan/bin/kudu with pid 27427
I20260926 15:12:31.603549 12427 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 27595
I20260926 15:12:31.616742 27913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:31.604272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33770 (local address 127.12.34.194:37165)
0926 15:12:31.604791 (+ 519us) server_negotiation.cc:206] Beginning negotiation
0926 15:12:31.604795 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:12:31.604812 (+ 17us) server_negotiation.cc:407] Connection header received
0926 15:12:31.604874 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:12:31.604878 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:12:31.604936 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:12:31.605023 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:12:31.606390 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:12:31.607278 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:12:31.608508 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:12:31.608948 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:12:31.609041 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:12:31.611648 (+ 2607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:12:31.611653 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:12:31.611657 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:12:31.611686 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:12:31.614379 (+ 2693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:12:31.615064 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:12:31.615069 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:12:31.615071 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:12:31.615191 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:12:31.615496 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:12:31.615499 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:12:31.615501 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:12:31.615749 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:12:31.615973 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0926 15:12:31.616182 (+ 209us) server_negotiation.cc:299] Negotiation successful
0926 15:12:31.616346 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424,"thread_start_us":306,"threads_started":1}
I20260926 15:12:31.617611 27718 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:12:31.700830 12427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8uoJGv/build/asan/bin/kudu with pid 27595
I20260926 15:12:31.721120 12427 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 27726
I20260926 15:12:31.743257 27915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:31.721995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52544 (local address 127.12.34.195:35357)
0926 15:12:31.722573 (+ 578us) server_negotiation.cc:206] Beginning negotiation
0926 15:12:31.722577 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:12:31.722593 (+ 16us) server_negotiation.cc:407] Connection header received
0926 15:12:31.722646 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:12:31.722649 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:12:31.722712 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:12:31.722837 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:12:31.724320 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:12:31.725067 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:12:31.726335 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:12:31.726747 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:12:31.726891 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:12:31.737823 (+ 10932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:12:31.737829 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:12:31.737832 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:12:31.737868 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:12:31.740578 (+ 2710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:12:31.741293 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:12:31.741312 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:12:31.741315 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:12:31.741451 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:12:31.741852 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:12:31.741857 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:12:31.741862 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:12:31.742122 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:12:31.742429 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0926 15:12:31.742661 (+ 232us) server_negotiation.cc:299] Negotiation successful
0926 15:12:31.742886 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":468,"thread_start_us":322,"threads_started":1}
I20260926 15:12:31.744109 27849 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:12:31.832614 12427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8uoJGv/build/asan/bin/kudu with pid 27726
2026-09-26T15:12:31Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (24216 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 15:12:31 dist-test-slave-865v krb5kdc[27919](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 15:12:31 dist-test-slave-865v krb5kdc[27919](info): set up 2 sockets
Sep 26 15:12:31 dist-test-slave-865v krb5kdc[27919](info): commencing operation
krb5kdc: starting...
W20260926 15:12:32.945645 12427 mini_kdc.cc:121] Time spent starting KDC: real 1.064s user 0.014s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/krb5kdc/test-user.keytab.
Sep 26 15:12:33 dist-test-slave-865v krb5kdc[27919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435553, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T15:12:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T15:12:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.34.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.34.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.34.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.254.keytab.
Entry for principal kudu/127.12.34.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.254.keytab.
I20260926 15:12:33.181020 12427 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8uoJGv/build/asan/bin/kudu
/tmp/dist-test-task8uoJGv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.34.254:45601
--webserver_interface=127.12.34.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.254.keytab
--principal=kudu/127.12.34.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.34.212:40893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.34.254:45601
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:12:33.500597 27932 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:12:33.500885 27932 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:12:33.501058 27932 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:12:33.508596 27932 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 15:12:33.508703 27932 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 15:12:33.508770 27932 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 15:12:33.508980 27932 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:12:33.509179 27932 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:12:33.509299 27932 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:12:33.509363 27932 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:12:33.509443 27932 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:12:33.519702 27932 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.34.212:40893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.12.34.254:45601
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.254.keytab
--principal=kudu/127.12.34.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.34.254:45601
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.34.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:12:33.521030 27932 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:12:33.522423 27932 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:12:33.535108 27938 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:12:33.535280 27937 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:12:33.535252 27940 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:12:33.535993 27932 server_base.cc:1034] running on GCE node
I20260926 15:12:33.536882 27932 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:12:33.538353 27932 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:12:33.539614 27932 hybrid_clock.cc:648] HybridClock initialized: now 1790435553539542 us; error 50 us; skew 500 ppm
I20260926 15:12:33.540021 27932 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:12:33 dist-test-slave-865v krb5kdc[27919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435553, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:12:33.543152 27932 init.cc:377] Logged in from keytab as kudu/127.12.34.254@KRBTEST.COM (short username kudu)
I20260926 15:12:33.544534 27932 webserver.cc:460] Webserver started at http://127.12.34.254:43841/ using document root <none> and password file <none>
I20260926 15:12:33.545251 27932 fs_manager.cc:359] Metadata directory not provided
I20260926 15:12:33.545347 27932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:12:33.545624 27932 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:12:33.548503 27932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/master-0/data/instance:
uuid: "7756b74c7007403c85a266f55693c0bc"
format_stamp: "Formatted at 2026-09-26 15:12:33 on dist-test-slave-865v"
server_key: "3adac6befea70a01de26d053623217b3"
server_key_iv: "eb96e0558a0965c6ed14ad592c0b6369"
server_key_version: "encryptionkey@0"
I20260926 15:12:33.549288 27932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/master-0/wal/instance:
uuid: "7756b74c7007403c85a266f55693c0bc"
format_stamp: "Formatted at 2026-09-26 15:12:33 on dist-test-slave-865v"
server_key: "3adac6befea70a01de26d053623217b3"
server_key_iv: "eb96e0558a0965c6ed14ad592c0b6369"
server_key_version: "encryptionkey@0"
I20260926 15:12:33.554311 27932 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 15:12:33.557752 27947 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:12:33.559167 27932 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 15:12:33.559315 27932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/master-0/data,/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/master-0/wal
uuid: "7756b74c7007403c85a266f55693c0bc"
format_stamp: "Formatted at 2026-09-26 15:12:33 on dist-test-slave-865v"
server_key: "3adac6befea70a01de26d053623217b3"
server_key_iv: "eb96e0558a0965c6ed14ad592c0b6369"
server_key_version: "encryptionkey@0"
I20260926 15:12:33.559482 27932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:12:33.573901 27932 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:12:33.578685 27932 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:12:33.578918 27932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:12:33.598289 27932 rpc_server.cc:307] RPC server started. Bound to: 127.12.34.254:45601
I20260926 15:12:33.598268 27998 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.34.254:45601 every 8 connection(s)
I20260926 15:12:33.599651 27932 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
I20260926 15:12:33.603370 27999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:12:33.604112 12427 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8uoJGv/build/asan/bin/kudu as pid 27932
I20260926 15:12:33.604421 12427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/master-0/wal/instance
I20260926 15:12:33.605357 12427 external_mini_cluster.cc:1428] Setting key 10f0ec94d48d202bf40cfa7948183d99
I20260926 15:12:33.619010 27999 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7756b74c7007403c85a266f55693c0bc: Bootstrap starting.
I20260926 15:12:33.622483 27999 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7756b74c7007403c85a266f55693c0bc: Neither blocks nor log segments found. Creating new log.
Sep 26 15:12:33 dist-test-slave-865v krb5kdc[27919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435553, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.34.254@KRBTEST.COM
I20260926 15:12:33.623579 27999 log.cc:824] T 00000000000000000000000000000000 P 7756b74c7007403c85a266f55693c0bc: Log is configured to *not* fsync() on all Append() calls
I20260926 15:12:33.627103 27999 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7756b74c7007403c85a266f55693c0bc: No bootstrap required, opened a new log
I20260926 15:12:33.633864 28002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:33.608168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49880 (local address 127.12.34.254:45601)
0926 15:12:33.609239 (+ 1071us) server_negotiation.cc:206] Beginning negotiation
0926 15:12:33.609256 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 15:12:33.609313 (+ 57us) server_negotiation.cc:407] Connection header received
0926 15:12:33.611117 (+ 1804us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:12:33.611146 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:12:33.611701 (+ 555us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:12:33.612154 (+ 453us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:12:33.614257 (+ 2103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:12:33.616678 (+ 2421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:12:33.618404 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:12:33.619473 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:12:33.619681 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:12:33.624706 (+ 5025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:12:33.624715 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:12:33.624730 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:12:33.624805 (+ 75us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:12:33.628921 (+ 4116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:12:33.629763 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:12:33.629771 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:12:33.629779 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:12:33.630017 (+ 238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:12:33.630406 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:12:33.630411 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:12:33.630415 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:12:33.631122 (+ 707us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:12:33.631445 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0926 15:12:33.631936 (+ 491us) server_negotiation.cc:299] Negotiation successful
0926 15:12:33.632431 (+ 495us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":754,"thread_start_us":508,"threads_started":1}
I20260926 15:12:33.637516 27999 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7756b74c7007403c85a266f55693c0bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7756b74c7007403c85a266f55693c0bc" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 45601 } }
I20260926 15:12:33.637849 27999 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7756b74c7007403c85a266f55693c0bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:12:33.637919 27999 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7756b74c7007403c85a266f55693c0bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7756b74c7007403c85a266f55693c0bc, State: Initialized, Role: FOLLOWER
I20260926 15:12:33.638442 27999 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7756b74c7007403c85a266f55693c0bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7756b74c7007403c85a266f55693c0bc" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 45601 } }
I20260926 15:12:33.638643 27999 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7756b74c7007403c85a266f55693c0bc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 15:12:33.638741 27999 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7756b74c7007403c85a266f55693c0bc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 15:12:33.638866 27999 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7756b74c7007403c85a266f55693c0bc [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:12:33.641297 27999 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7756b74c7007403c85a266f55693c0bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7756b74c7007403c85a266f55693c0bc" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 45601 } }
I20260926 15:12:33.641904 27999 leader_election.cc:302] T 00000000000000000000000000000000 P 7756b74c7007403c85a266f55693c0bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7756b74c7007403c85a266f55693c0bc; no voters:
I20260926 15:12:33.642973 27999 leader_election.cc:288] T 00000000000000000000000000000000 P 7756b74c7007403c85a266f55693c0bc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 15:12:33.642987 28004 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7756b74c7007403c85a266f55693c0bc [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:12:33.644363 28004 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7756b74c7007403c85a266f55693c0bc [term 1 LEADER]: Becoming Leader. State: Replica: 7756b74c7007403c85a266f55693c0bc, State: Running, Role: LEADER
I20260926 15:12:33.645167 28004 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7756b74c7007403c85a266f55693c0bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7756b74c7007403c85a266f55693c0bc" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 45601 } }
I20260926 15:12:33.646509 27999 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7756b74c7007403c85a266f55693c0bc [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:12:33.648507 28006 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7756b74c7007403c85a266f55693c0bc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7756b74c7007403c85a266f55693c0bc. Latest consensus state: current_term: 1 leader_uuid: "7756b74c7007403c85a266f55693c0bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7756b74c7007403c85a266f55693c0bc" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 45601 } } }
I20260926 15:12:33.648422 28005 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7756b74c7007403c85a266f55693c0bc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7756b74c7007403c85a266f55693c0bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7756b74c7007403c85a266f55693c0bc" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 45601 } } }
I20260926 15:12:33.648800 28006 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7756b74c7007403c85a266f55693c0bc [sys.catalog]: This master's current role is: LEADER
I20260926 15:12:33.648809 28005 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7756b74c7007403c85a266f55693c0bc [sys.catalog]: This master's current role is: LEADER
I20260926 15:12:33.649735 28009 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:12:33.655072 28009 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 15:12:33.666908 28009 catalog_manager.cc:1347] Generated new cluster ID: dd2c37d372564116a5191e490b674674
I20260926 15:12:33.667027 28009 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 15:12:33.676373 28009 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 15:12:33.678159 28009 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:12:33.690279 28009 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7756b74c7007403c85a266f55693c0bc: Generated new TSK 0
I20260926 15:12:33.691376 28009 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.34.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.34.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.34.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.193.keytab.
Entry for principal kudu/127.12.34.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.193.keytab.
I20260926 15:12:33.757572 12427 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8uoJGv/build/asan/bin/kudu
/tmp/dist-test-task8uoJGv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.34.193:0
--local_ip_for_outbound_sockets=127.12.34.193
--webserver_interface=127.12.34.193
--webserver_port=0
--tserver_master_addrs=127.12.34.254:45601
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.193.keytab
--principal=kudu/127.12.34.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.34.212:40893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:12:34.083051 28025 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:12:34.083353 28025 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:12:34.083518 28025 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:12:34.090824 28025 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:12:34.090960 28025 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:12:34.091141 28025 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.34.193
I20260926 15:12:34.100451 28025 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.34.212:40893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.193.keytab
--principal=kudu/127.12.34.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.34.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.34.193
--webserver_port=0
--tserver_master_addrs=127.12.34.254:45601
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.34.193
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:12:34.102010 28025 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:12:34.103217 28025 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:12:34.115551 28030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:12:34.115586 28031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:12:34.115640 28033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:12:34.116567 28025 server_base.cc:1034] running on GCE node
I20260926 15:12:34.117178 28025 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:12:34.118088 28025 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:12:34.119283 28025 hybrid_clock.cc:648] HybridClock initialized: now 1790435554119245 us; error 32 us; skew 500 ppm
I20260926 15:12:34.119555 28025 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:12:34 dist-test-slave-865v krb5kdc[27919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435554, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:12:34.122669 28025 init.cc:377] Logged in from keytab as kudu/127.12.34.193@KRBTEST.COM (short username kudu)
I20260926 15:12:34.124239 28025 webserver.cc:460] Webserver started at http://127.12.34.193:36813/ using document root <none> and password file <none>
I20260926 15:12:34.124890 28025 fs_manager.cc:359] Metadata directory not provided
I20260926 15:12:34.125056 28025 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:12:34.125387 28025 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:12:34.128872 28025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/ts-0/data/instance:
uuid: "a311dd73aa814f63b863f901521cdcea"
format_stamp: "Formatted at 2026-09-26 15:12:34 on dist-test-slave-865v"
server_key: "c274070bb383f9b2ce7d764574679349"
server_key_iv: "6e617dfd8ed626a53b3b93bba0428b34"
server_key_version: "encryptionkey@0"
I20260926 15:12:34.129710 28025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/ts-0/wal/instance:
uuid: "a311dd73aa814f63b863f901521cdcea"
format_stamp: "Formatted at 2026-09-26 15:12:34 on dist-test-slave-865v"
server_key: "c274070bb383f9b2ce7d764574679349"
server_key_iv: "6e617dfd8ed626a53b3b93bba0428b34"
server_key_version: "encryptionkey@0"
I20260926 15:12:34.134682 28025 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 15:12:34.137995 28040 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:12:34.139225 28025 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 15:12:34.139405 28025 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/ts-0/data,/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/ts-0/wal
uuid: "a311dd73aa814f63b863f901521cdcea"
format_stamp: "Formatted at 2026-09-26 15:12:34 on dist-test-slave-865v"
server_key: "c274070bb383f9b2ce7d764574679349"
server_key_iv: "6e617dfd8ed626a53b3b93bba0428b34"
server_key_version: "encryptionkey@0"
I20260926 15:12:34.139618 28025 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:12:34.158656 28025 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:12:34.162410 28025 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:12:34.162689 28025 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:12:34.163712 28025 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:12:34.165043 28025 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:12:34.165158 28025 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:12:34.165307 28025 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:12:34.165380 28025 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:12:34.197736 28025 rpc_server.cc:307] RPC server started. Bound to: 127.12.34.193:45901
I20260926 15:12:34.197818 28152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.34.193:45901 every 8 connection(s)
I20260926 15:12:34.198963 28025 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/ts-0/data/info.pb
I20260926 15:12:34.202409 12427 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8uoJGv/build/asan/bin/kudu as pid 28025
I20260926 15:12:34.202651 12427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/ts-0/wal/instance
I20260926 15:12:34.203378 12427 external_mini_cluster.cc:1428] Setting key e85e2d2199a9d398e4575c6f5e4db963
Sep 26 15:12:34 dist-test-slave-865v krb5kdc[27919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435554, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.193@KRBTEST.COM for kudu/127.12.34.254@KRBTEST.COM
I20260926 15:12:34.219457 28155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:34.201466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.193:33421 (local address 127.12.34.254:45601)
0926 15:12:34.201892 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0926 15:12:34.201896 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:12:34.202958 (+ 1062us) server_negotiation.cc:407] Connection header received
0926 15:12:34.204919 (+ 1961us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:12:34.204931 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:12:34.204989 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:12:34.205149 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:12:34.206573 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:12:34.207302 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:12:34.208764 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:12:34.209237 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:12:34.209375 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:12:34.212680 (+ 3305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:12:34.212689 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:12:34.212697 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:12:34.212742 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:12:34.216268 (+ 3526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:12:34.216949 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:12:34.216957 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:12:34.216963 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:12:34.217130 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:12:34.217525 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:12:34.217534 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:12:34.217540 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:12:34.217920 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:12:34.218060 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0926 15:12:34.218697 (+ 637us) server_negotiation.cc:299] Negotiation successful
0926 15:12:34.218981 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":233,"threads_started":1}
I20260926 15:12:34.220036 28156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:34.201976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:45601 (local address 127.12.34.193:33421)
0926 15:12:34.202794 (+ 818us) negotiation.cc:107] Waiting for socket to connect
0926 15:12:34.202836 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 15:12:34.204647 (+ 1811us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:12:34.205358 (+ 711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:12:34.205373 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:12:34.205839 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:12:34.206338 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:12:34.206388 (+ 50us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:12:34.207464 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:12:34.207469 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:12:34.208621 (+ 1152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:12:34.208633 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:12:34.209469 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:12:34.209478 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:12:34.209700 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:12:34.210359 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:12:34.210392 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:12:34.212479 (+ 2087us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:12:34.216407 (+ 3928us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:12:34.216413 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:12:34.216429 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:12:34.216820 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:12:34.217238 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:12:34.217241 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:12:34.217243 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:12:34.217409 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:12:34.218044 (+ 635us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:12:34.218049 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:12:34.218333 (+ 284us) client_negotiation.cc:769] Sending connection context
0926 15:12:34.218559 (+ 226us) client_negotiation.cc:240] Negotiation successful
0926 15:12:34.218907 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":336,"threads_started":1}
I20260926 15:12:34.221549 28153 heartbeater.cc:347] Connected to a master server at 127.12.34.254:45601
I20260926 15:12:34.221846 28153 heartbeater.cc:464] Registering TS with master...
I20260926 15:12:34.222534 28153 heartbeater.cc:511] Master 127.12.34.254:45601 requested a full tablet report, sending...
I20260926 15:12:34.224502 27964 ts_manager.cc:194] Registered new tserver with Master: a311dd73aa814f63b863f901521cdcea (127.12.34.193:45901)
I20260926 15:12:34.225941 27964 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.34.193@KRBTEST.COM'} at 127.12.34.193:33421
WARNING: no policy specified for kudu/127.12.34.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.34.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.34.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.194.keytab.
Entry for principal kudu/127.12.34.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.194.keytab.
I20260926 15:12:34.256724 12427 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8uoJGv/build/asan/bin/kudu
/tmp/dist-test-task8uoJGv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.34.194:0
--local_ip_for_outbound_sockets=127.12.34.194
--webserver_interface=127.12.34.194
--webserver_port=0
--tserver_master_addrs=127.12.34.254:45601
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.194.keytab
--principal=kudu/127.12.34.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.34.212:40893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:12:34.573205 28159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:12:34.573589 28159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:12:34.573769 28159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:12:34.582386 28159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:12:34.582492 28159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:12:34.582705 28159 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.34.194
I20260926 15:12:34.592564 28159 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.34.212:40893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.194.keytab
--principal=kudu/127.12.34.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.34.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.34.194
--webserver_port=0
--tserver_master_addrs=127.12.34.254:45601
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.34.194
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:12:34.593784 28159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:12:34.594992 28159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:12:34.609608 28164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:12:34.609686 28165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:12:34.609805 28167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:12:34.610831 28159 server_base.cc:1034] running on GCE node
I20260926 15:12:34.611400 28159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:12:34.612342 28159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:12:34.613590 28159 hybrid_clock.cc:648] HybridClock initialized: now 1790435554613544 us; error 34 us; skew 500 ppm
I20260926 15:12:34.613927 28159 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:12:34 dist-test-slave-865v krb5kdc[27919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435554, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:12:34.617010 28159 init.cc:377] Logged in from keytab as kudu/127.12.34.194@KRBTEST.COM (short username kudu)
I20260926 15:12:34.618507 28159 webserver.cc:460] Webserver started at http://127.12.34.194:36799/ using document root <none> and password file <none>
I20260926 15:12:34.619179 28159 fs_manager.cc:359] Metadata directory not provided
I20260926 15:12:34.619324 28159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:12:34.619622 28159 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:12:34.622423 28159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/ts-1/data/instance:
uuid: "9fda66f1b152460e8e0cb199f028a5c0"
format_stamp: "Formatted at 2026-09-26 15:12:34 on dist-test-slave-865v"
server_key: "fd39586b7675ea818ec3df58e8cc92a9"
server_key_iv: "d6367cc0dc316147b92055a8d8fed163"
server_key_version: "encryptionkey@0"
I20260926 15:12:34.623169 28159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/ts-1/wal/instance:
uuid: "9fda66f1b152460e8e0cb199f028a5c0"
format_stamp: "Formatted at 2026-09-26 15:12:34 on dist-test-slave-865v"
server_key: "fd39586b7675ea818ec3df58e8cc92a9"
server_key_iv: "d6367cc0dc316147b92055a8d8fed163"
server_key_version: "encryptionkey@0"
I20260926 15:12:34.628007 28159 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260926 15:12:34.631348 28174 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:12:34.632594 28159 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260926 15:12:34.632767 28159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/ts-1/data,/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/ts-1/wal
uuid: "9fda66f1b152460e8e0cb199f028a5c0"
format_stamp: "Formatted at 2026-09-26 15:12:34 on dist-test-slave-865v"
server_key: "fd39586b7675ea818ec3df58e8cc92a9"
server_key_iv: "d6367cc0dc316147b92055a8d8fed163"
server_key_version: "encryptionkey@0"
I20260926 15:12:34.632952 28159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:12:34.659478 28159 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:12:34.664093 28159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:12:34.664390 28159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:12:34.665407 28159 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:12:34.666782 28159 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:12:34.666887 28159 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:12:34.667039 28159 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:12:34.667114 28159 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:12:34.702296 28159 rpc_server.cc:307] RPC server started. Bound to: 127.12.34.194:38857
I20260926 15:12:34.702383 28286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.34.194:38857 every 8 connection(s)
I20260926 15:12:34.703572 28159 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/ts-1/data/info.pb
I20260926 15:12:34.708954 12427 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8uoJGv/build/asan/bin/kudu as pid 28159
I20260926 15:12:34.709178 12427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/ts-1/wal/instance
I20260926 15:12:34.709677 12427 external_mini_cluster.cc:1428] Setting key d71372415c5fc0aba4e9f572c2e6b883
Sep 26 15:12:34 dist-test-slave-865v krb5kdc[27919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435554, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.194@KRBTEST.COM for kudu/127.12.34.254@KRBTEST.COM
I20260926 15:12:34.724834 28155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:34.706313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.194:42083 (local address 127.12.34.254:45601)
0926 15:12:34.706531 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0926 15:12:34.706537 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:12:34.707757 (+ 1220us) server_negotiation.cc:407] Connection header received
0926 15:12:34.709893 (+ 2136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:12:34.709900 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:12:34.709994 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:12:34.710110 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:12:34.711788 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:12:34.712548 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:12:34.714268 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:12:34.714732 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:12:34.714879 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:12:34.719021 (+ 4142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:12:34.719026 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:12:34.719029 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:12:34.719060 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:12:34.721672 (+ 2612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:12:34.722599 (+ 927us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:12:34.722603 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:12:34.722605 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:12:34.722721 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:12:34.723198 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:12:34.723201 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:12:34.723203 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:12:34.723464 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:12:34.723574 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0926 15:12:34.724440 (+ 866us) server_negotiation.cc:299] Negotiation successful
0926 15:12:34.724601 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260926 15:12:34.726327 28289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:34.706807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:45601 (local address 127.12.34.194:42083)
0926 15:12:34.707580 (+ 773us) negotiation.cc:107] Waiting for socket to connect
0926 15:12:34.707642 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0926 15:12:34.709584 (+ 1942us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:12:34.710340 (+ 756us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:12:34.710348 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:12:34.711006 (+ 658us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:12:34.711618 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:12:34.711635 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:12:34.712718 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:12:34.712724 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:12:34.714090 (+ 1366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:12:34.714107 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:12:34.714978 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:12:34.714986 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:12:34.715222 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:12:34.715950 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:12:34.715974 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:12:34.718753 (+ 2779us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:12:34.721825 (+ 3072us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:12:34.721836 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:12:34.721854 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:12:34.722469 (+ 615us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:12:34.722825 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:12:34.722832 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:12:34.722839 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:12:34.723073 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:12:34.723594 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:12:34.723605 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:12:34.724038 (+ 433us) client_negotiation.cc:769] Sending connection context
0926 15:12:34.724322 (+ 284us) client_negotiation.cc:240] Negotiation successful
0926 15:12:34.724875 (+ 553us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":285,"threads_started":1}
I20260926 15:12:34.728022 28287 heartbeater.cc:347] Connected to a master server at 127.12.34.254:45601
I20260926 15:12:34.728494 28287 heartbeater.cc:464] Registering TS with master...
I20260926 15:12:34.729344 28287 heartbeater.cc:511] Master 127.12.34.254:45601 requested a full tablet report, sending...
I20260926 15:12:34.731123 27964 ts_manager.cc:194] Registered new tserver with Master: 9fda66f1b152460e8e0cb199f028a5c0 (127.12.34.194:38857)
I20260926 15:12:34.731993 27964 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.34.194@KRBTEST.COM'} at 127.12.34.194:42083
WARNING: no policy specified for kudu/127.12.34.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.34.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.34.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.195.keytab.
Entry for principal kudu/127.12.34.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.195.keytab.
I20260926 15:12:34.765396 12427 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8uoJGv/build/asan/bin/kudu
/tmp/dist-test-task8uoJGv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.34.195:0
--local_ip_for_outbound_sockets=127.12.34.195
--webserver_interface=127.12.34.195
--webserver_port=0
--tserver_master_addrs=127.12.34.254:45601
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.195.keytab
--principal=kudu/127.12.34.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.34.212:40893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:12:35.080183 28292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:12:35.080518 28292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:12:35.080691 28292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:12:35.088061 28292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:12:35.088220 28292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:12:35.088400 28292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.34.195
I20260926 15:12:35.098057 28292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.34.212:40893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.195.keytab
--principal=kudu/127.12.34.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.34.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.34.195
--webserver_port=0
--tserver_master_addrs=127.12.34.254:45601
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.34.195
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:12:35.099314 28292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:12:35.100577 28292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:12:35.113381 28300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:12:35.113442 28297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:12:35.113464 28298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:12:35.114454 28292 server_base.cc:1034] running on GCE node
I20260926 15:12:35.115020 28292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:12:35.115895 28292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:12:35.117214 28292 hybrid_clock.cc:648] HybridClock initialized: now 1790435555117176 us; error 54 us; skew 500 ppm
I20260926 15:12:35.117554 28292 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:12:35 dist-test-slave-865v krb5kdc[27919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435555, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:12:35.120625 28292 init.cc:377] Logged in from keytab as kudu/127.12.34.195@KRBTEST.COM (short username kudu)
I20260926 15:12:35.122120 28292 webserver.cc:460] Webserver started at http://127.12.34.195:33713/ using document root <none> and password file <none>
I20260926 15:12:35.122795 28292 fs_manager.cc:359] Metadata directory not provided
I20260926 15:12:35.122954 28292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:12:35.123265 28292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:12:35.126192 28292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/ts-2/data/instance:
uuid: "6e3b1a3f8e1343239a873c89195df725"
format_stamp: "Formatted at 2026-09-26 15:12:35 on dist-test-slave-865v"
server_key: "5f8da085443f5841cf5692d98bb00261"
server_key_iv: "d68a194c90806156ba2e288e7d369e21"
server_key_version: "encryptionkey@0"
I20260926 15:12:35.126971 28292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/ts-2/wal/instance:
uuid: "6e3b1a3f8e1343239a873c89195df725"
format_stamp: "Formatted at 2026-09-26 15:12:35 on dist-test-slave-865v"
server_key: "5f8da085443f5841cf5692d98bb00261"
server_key_iv: "d68a194c90806156ba2e288e7d369e21"
server_key_version: "encryptionkey@0"
I20260926 15:12:35.132242 28292 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260926 15:12:35.136799 28307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:12:35.138175 28292 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 15:12:35.138356 28292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/ts-2/data,/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/ts-2/wal
uuid: "6e3b1a3f8e1343239a873c89195df725"
format_stamp: "Formatted at 2026-09-26 15:12:35 on dist-test-slave-865v"
server_key: "5f8da085443f5841cf5692d98bb00261"
server_key_iv: "d68a194c90806156ba2e288e7d369e21"
server_key_version: "encryptionkey@0"
I20260926 15:12:35.138571 28292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:12:35.151010 28292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:12:35.154639 28292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:12:35.154918 28292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:12:35.155815 28292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:12:35.157166 28292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:12:35.157271 28292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:12:35.157426 28292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:12:35.157506 28292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:12:35.190163 28292 rpc_server.cc:307] RPC server started. Bound to: 127.12.34.195:44121
I20260926 15:12:35.190207 28419 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.34.195:44121 every 8 connection(s)
I20260926 15:12:35.191390 28292 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/ts-2/data/info.pb
I20260926 15:12:35.198048 12427 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8uoJGv/build/asan/bin/kudu as pid 28292
I20260926 15:12:35.198277 12427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/ts-2/wal/instance
I20260926 15:12:35.198747 12427 external_mini_cluster.cc:1428] Setting key 75a78aaf6e15726be57cb8f3a19a284b
Sep 26 15:12:35 dist-test-slave-865v krb5kdc[27919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435555, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.195@KRBTEST.COM for kudu/127.12.34.254@KRBTEST.COM
I20260926 15:12:35.211630 28155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:35.193958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.195:57357 (local address 127.12.34.254:45601)
0926 15:12:35.194138 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0926 15:12:35.194143 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:12:35.195169 (+ 1026us) server_negotiation.cc:407] Connection header received
0926 15:12:35.196855 (+ 1686us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:12:35.196859 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:12:35.196939 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:12:35.197028 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:12:35.198989 (+ 1961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:12:35.200136 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:12:35.202436 (+ 2300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:12:35.202838 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:12:35.202955 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:12:35.206257 (+ 3302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:12:35.206262 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:12:35.206265 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:12:35.206295 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:12:35.208945 (+ 2650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:12:35.209684 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:12:35.209688 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:12:35.209691 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:12:35.209852 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:12:35.210258 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:12:35.210262 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:12:35.210265 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:12:35.210539 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:12:35.210649 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0926 15:12:35.211219 (+ 570us) server_negotiation.cc:299] Negotiation successful
0926 15:12:35.211375 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 15:12:35.212441 28422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:35.194354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:45601 (local address 127.12.34.195:57357)
0926 15:12:35.195011 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0926 15:12:35.195053 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 15:12:35.196612 (+ 1559us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:12:35.197323 (+ 711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:12:35.197335 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:12:35.198045 (+ 710us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:12:35.198798 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:12:35.198822 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:12:35.200334 (+ 1512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:12:35.200340 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:12:35.202272 (+ 1932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:12:35.202286 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:12:35.203050 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:12:35.203058 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:12:35.203323 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:12:35.203984 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:12:35.204007 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:12:35.206081 (+ 2074us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:12:35.209136 (+ 3055us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:12:35.209142 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:12:35.209152 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:12:35.209539 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:12:35.209970 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:12:35.209973 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:12:35.209976 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:12:35.210134 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:12:35.210641 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:12:35.210646 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:12:35.210905 (+ 259us) client_negotiation.cc:769] Sending connection context
0926 15:12:35.211100 (+ 195us) client_negotiation.cc:240] Negotiation successful
0926 15:12:35.211426 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":265,"threads_started":1}
I20260926 15:12:35.213794 28420 heartbeater.cc:347] Connected to a master server at 127.12.34.254:45601
I20260926 15:12:35.214130 28420 heartbeater.cc:464] Registering TS with master...
I20260926 15:12:35.214857 28420 heartbeater.cc:511] Master 127.12.34.254:45601 requested a full tablet report, sending...
I20260926 15:12:35.216138 27964 ts_manager.cc:194] Registered new tserver with Master: 6e3b1a3f8e1343239a873c89195df725 (127.12.34.195:44121)
I20260926 15:12:35.216921 27964 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.34.195@KRBTEST.COM'} at 127.12.34.195:57357
I20260926 15:12:35.226358 12427 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 15:12:35.228648 28153 heartbeater.cc:503] Master 127.12.34.254:45601 was elected leader, sending a full tablet report...
I20260926 15:12:35.243456 28155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:35.230982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49888 (local address 127.12.34.254:45601)
0926 15:12:35.231130 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0926 15:12:35.231134 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:12:35.231592 (+ 458us) server_negotiation.cc:407] Connection header received
0926 15:12:35.231792 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:12:35.231798 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:12:35.231865 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:12:35.231972 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:12:35.233461 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:12:35.234160 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:12:35.235533 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:12:35.235956 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:12:35.236111 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:12:35.238496 (+ 2385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:12:35.238500 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:12:35.238503 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:12:35.238532 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:12:35.240769 (+ 2237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:12:35.241464 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:12:35.241471 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:12:35.241474 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:12:35.241598 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:12:35.241986 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:12:35.241991 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:12:35.241997 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:12:35.242302 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:12:35.242560 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0926 15:12:35.242904 (+ 344us) server_negotiation.cc:299] Negotiation successful
0926 15:12:35.243115 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 15:12:35.247362 27964 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49888:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 15:12:35.254072 27964 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 15:12:35.275424 28432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:35.266005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.195:44121 (local address 127.0.0.1:40196)
0926 15:12:35.266672 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0926 15:12:35.266718 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 15:12:35.266931 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:12:35.267770 (+ 839us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:12:35.267778 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:12:35.267807 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:12:35.268301 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:12:35.268315 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:12:35.270367 (+ 2052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:12:35.270371 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:12:35.272736 (+ 2365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:12:35.272748 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:12:35.274516 (+ 1768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:12:35.274522 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:12:35.274706 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:12:35.274758 (+ 52us) client_negotiation.cc:769] Sending connection context
0926 15:12:35.274890 (+ 132us) client_negotiation.cc:240] Negotiation successful
0926 15:12:35.275063 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":298,"threads_started":1}
I20260926 15:12:35.275424 28433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:35.266232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40196 (local address 127.12.34.195:44121)
0926 15:12:35.267292 (+ 1060us) server_negotiation.cc:206] Beginning negotiation
0926 15:12:35.267298 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:12:35.267322 (+ 24us) server_negotiation.cc:407] Connection header received
0926 15:12:35.267410 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:12:35.267415 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:12:35.267582 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:12:35.267763 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:12:35.268519 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:12:35.270206 (+ 1687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:12:35.272936 (+ 2730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:12:35.274276 (+ 1340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:12:35.274401 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:12:35.274614 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0926 15:12:35.274959 (+ 345us) server_negotiation.cc:299] Negotiation successful
0926 15:12:35.275114 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":926,"thread_start_us":772,"threads_started":1}
I20260926 15:12:35.276825 28434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:35.266619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.194:38857 (local address 127.0.0.1:51178)
0926 15:12:35.268430 (+ 1811us) negotiation.cc:107] Waiting for socket to connect
0926 15:12:35.268448 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 15:12:35.268542 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:12:35.269739 (+ 1197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:12:35.269744 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:12:35.269767 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:12:35.270128 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:12:35.270138 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:12:35.272176 (+ 2038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:12:35.272185 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:12:35.274270 (+ 2085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:12:35.274280 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:12:35.276015 (+ 1735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:12:35.276022 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:12:35.276185 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:12:35.276214 (+ 29us) client_negotiation.cc:769] Sending connection context
0926 15:12:35.276312 (+ 98us) client_negotiation.cc:240] Negotiation successful
0926 15:12:35.276442 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1670,"thread_start_us":1513,"threads_started":1}
I20260926 15:12:35.277495 28435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:35.266850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51178 (local address 127.12.34.194:38857)
0926 15:12:35.269220 (+ 2370us) server_negotiation.cc:206] Beginning negotiation
0926 15:12:35.269228 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 15:12:35.269252 (+ 24us) server_negotiation.cc:407] Connection header received
0926 15:12:35.269334 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:12:35.269341 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:12:35.269544 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:12:35.270338 (+ 794us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:12:35.270568 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:12:35.272028 (+ 1460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:12:35.274410 (+ 2382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:12:35.275803 (+ 1393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:12:35.275940 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:12:35.276133 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0926 15:12:35.277131 (+ 998us) server_negotiation.cc:299] Negotiation successful
0926 15:12:35.277290 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2169,"thread_start_us":2068,"threads_started":1}
I20260926 15:12:35.279588 28355 tablet_service.cc:1467] Processing CreateTablet for tablet b78c1e264d414455ab3f8e9996042a24 (DEFAULT_TABLE table=test-table [id=0c1066636dff487283a72186f69cb0b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:12:35.280229 28436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:35.268577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.193:45901 (local address 127.0.0.1:43264)
0926 15:12:35.272611 (+ 4034us) negotiation.cc:107] Waiting for socket to connect
0926 15:12:35.272628 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 15:12:35.272749 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:12:35.273417 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:12:35.273422 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:12:35.273446 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:12:35.273821 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:12:35.273831 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:12:35.275566 (+ 1735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:12:35.275573 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:12:35.277617 (+ 2044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:12:35.277631 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:12:35.279479 (+ 1848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:12:35.279485 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:12:35.279659 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:12:35.279698 (+ 39us) client_negotiation.cc:769] Sending connection context
0926 15:12:35.279792 (+ 94us) client_negotiation.cc:240] Negotiation successful
0926 15:12:35.279910 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3894,"thread_start_us":3738,"threads_started":1}
I20260926 15:12:35.280941 28437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:35.269140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43264 (local address 127.12.34.193:45901)
0926 15:12:35.269786 (+ 646us) server_negotiation.cc:206] Beginning negotiation
0926 15:12:35.269798 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 15:12:35.272888 (+ 3090us) server_negotiation.cc:407] Connection header received
0926 15:12:35.272987 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:12:35.272997 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:12:35.273224 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:12:35.273419 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:12:35.273983 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:12:35.275371 (+ 1388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:12:35.277789 (+ 2418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:12:35.279255 (+ 1466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:12:35.279408 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:12:35.279623 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0926 15:12:35.280451 (+ 828us) server_negotiation.cc:299] Negotiation successful
0926 15:12:35.280614 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":456,"thread_start_us":306,"threads_started":1}
I20260926 15:12:35.281167 28355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b78c1e264d414455ab3f8e9996042a24. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:12:35.281016 28222 tablet_service.cc:1467] Processing CreateTablet for tablet b78c1e264d414455ab3f8e9996042a24 (DEFAULT_TABLE table=test-table [id=0c1066636dff487283a72186f69cb0b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:12:35.282384 28222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b78c1e264d414455ab3f8e9996042a24. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:12:35.284466 28088 tablet_service.cc:1467] Processing CreateTablet for tablet b78c1e264d414455ab3f8e9996042a24 (DEFAULT_TABLE table=test-table [id=0c1066636dff487283a72186f69cb0b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:12:35.285787 28088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b78c1e264d414455ab3f8e9996042a24. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:12:35.292578 28438 tablet_bootstrap.cc:492] T b78c1e264d414455ab3f8e9996042a24 P 6e3b1a3f8e1343239a873c89195df725: Bootstrap starting.
I20260926 15:12:35.295949 28438 tablet_bootstrap.cc:654] T b78c1e264d414455ab3f8e9996042a24 P 6e3b1a3f8e1343239a873c89195df725: Neither blocks nor log segments found. Creating new log.
I20260926 15:12:35.296342 28439 tablet_bootstrap.cc:492] T b78c1e264d414455ab3f8e9996042a24 P 9fda66f1b152460e8e0cb199f028a5c0: Bootstrap starting.
I20260926 15:12:35.297348 28438 log.cc:824] T b78c1e264d414455ab3f8e9996042a24 P 6e3b1a3f8e1343239a873c89195df725: Log is configured to *not* fsync() on all Append() calls
I20260926 15:12:35.299281 28441 tablet_bootstrap.cc:492] T b78c1e264d414455ab3f8e9996042a24 P a311dd73aa814f63b863f901521cdcea: Bootstrap starting.
I20260926 15:12:35.299589 28439 tablet_bootstrap.cc:654] T b78c1e264d414455ab3f8e9996042a24 P 9fda66f1b152460e8e0cb199f028a5c0: Neither blocks nor log segments found. Creating new log.
I20260926 15:12:35.300769 28438 tablet_bootstrap.cc:492] T b78c1e264d414455ab3f8e9996042a24 P 6e3b1a3f8e1343239a873c89195df725: No bootstrap required, opened a new log
I20260926 15:12:35.300827 28439 log.cc:824] T b78c1e264d414455ab3f8e9996042a24 P 9fda66f1b152460e8e0cb199f028a5c0: Log is configured to *not* fsync() on all Append() calls
I20260926 15:12:35.301033 28438 ts_tablet_manager.cc:1397] T b78c1e264d414455ab3f8e9996042a24 P 6e3b1a3f8e1343239a873c89195df725: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260926 15:12:35.302284 28441 tablet_bootstrap.cc:654] T b78c1e264d414455ab3f8e9996042a24 P a311dd73aa814f63b863f901521cdcea: Neither blocks nor log segments found. Creating new log.
I20260926 15:12:35.303476 28441 log.cc:824] T b78c1e264d414455ab3f8e9996042a24 P a311dd73aa814f63b863f901521cdcea: Log is configured to *not* fsync() on all Append() calls
I20260926 15:12:35.303715 28439 tablet_bootstrap.cc:492] T b78c1e264d414455ab3f8e9996042a24 P 9fda66f1b152460e8e0cb199f028a5c0: No bootstrap required, opened a new log
I20260926 15:12:35.303968 28439 ts_tablet_manager.cc:1397] T b78c1e264d414455ab3f8e9996042a24 P 9fda66f1b152460e8e0cb199f028a5c0: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260926 15:12:35.306318 28441 tablet_bootstrap.cc:492] T b78c1e264d414455ab3f8e9996042a24 P a311dd73aa814f63b863f901521cdcea: No bootstrap required, opened a new log
I20260926 15:12:35.306555 28441 ts_tablet_manager.cc:1397] T b78c1e264d414455ab3f8e9996042a24 P a311dd73aa814f63b863f901521cdcea: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260926 15:12:35.309362 28438 raft_consensus.cc:348] T b78c1e264d414455ab3f8e9996042a24 P 6e3b1a3f8e1343239a873c89195df725 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e3b1a3f8e1343239a873c89195df725" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 44121 } } peers { permanent_uuid: "9fda66f1b152460e8e0cb199f028a5c0" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 38857 } } peers { permanent_uuid: "a311dd73aa814f63b863f901521cdcea" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 45901 } }
I20260926 15:12:35.309713 28438 raft_consensus.cc:374] T b78c1e264d414455ab3f8e9996042a24 P 6e3b1a3f8e1343239a873c89195df725 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:12:35.309813 28438 raft_consensus.cc:729] T b78c1e264d414455ab3f8e9996042a24 P 6e3b1a3f8e1343239a873c89195df725 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e3b1a3f8e1343239a873c89195df725, State: Initialized, Role: FOLLOWER
I20260926 15:12:35.310472 28438 consensus_queue.cc:260] T b78c1e264d414455ab3f8e9996042a24 P 6e3b1a3f8e1343239a873c89195df725 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e3b1a3f8e1343239a873c89195df725" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 44121 } } peers { permanent_uuid: "9fda66f1b152460e8e0cb199f028a5c0" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 38857 } } peers { permanent_uuid: "a311dd73aa814f63b863f901521cdcea" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 45901 } }
I20260926 15:12:35.313750 28420 heartbeater.cc:503] Master 127.12.34.254:45601 was elected leader, sending a full tablet report...
I20260926 15:12:35.314203 28439 raft_consensus.cc:348] T b78c1e264d414455ab3f8e9996042a24 P 9fda66f1b152460e8e0cb199f028a5c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e3b1a3f8e1343239a873c89195df725" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 44121 } } peers { permanent_uuid: "9fda66f1b152460e8e0cb199f028a5c0" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 38857 } } peers { permanent_uuid: "a311dd73aa814f63b863f901521cdcea" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 45901 } }
I20260926 15:12:35.314616 28439 raft_consensus.cc:374] T b78c1e264d414455ab3f8e9996042a24 P 9fda66f1b152460e8e0cb199f028a5c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:12:35.314706 28439 raft_consensus.cc:729] T b78c1e264d414455ab3f8e9996042a24 P 9fda66f1b152460e8e0cb199f028a5c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fda66f1b152460e8e0cb199f028a5c0, State: Initialized, Role: FOLLOWER
I20260926 15:12:35.315443 28439 consensus_queue.cc:260] T b78c1e264d414455ab3f8e9996042a24 P 9fda66f1b152460e8e0cb199f028a5c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e3b1a3f8e1343239a873c89195df725" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 44121 } } peers { permanent_uuid: "9fda66f1b152460e8e0cb199f028a5c0" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 38857 } } peers { permanent_uuid: "a311dd73aa814f63b863f901521cdcea" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 45901 } }
I20260926 15:12:35.315676 28438 ts_tablet_manager.cc:1428] T b78c1e264d414455ab3f8e9996042a24 P 6e3b1a3f8e1343239a873c89195df725: Time spent starting tablet: real 0.015s user 0.010s sys 0.003s
I20260926 15:12:35.316589 28287 heartbeater.cc:503] Master 127.12.34.254:45601 was elected leader, sending a full tablet report...
I20260926 15:12:35.316892 28441 raft_consensus.cc:348] T b78c1e264d414455ab3f8e9996042a24 P a311dd73aa814f63b863f901521cdcea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e3b1a3f8e1343239a873c89195df725" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 44121 } } peers { permanent_uuid: "9fda66f1b152460e8e0cb199f028a5c0" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 38857 } } peers { permanent_uuid: "a311dd73aa814f63b863f901521cdcea" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 45901 } }
I20260926 15:12:35.317220 28439 ts_tablet_manager.cc:1428] T b78c1e264d414455ab3f8e9996042a24 P 9fda66f1b152460e8e0cb199f028a5c0: Time spent starting tablet: real 0.013s user 0.008s sys 0.005s
I20260926 15:12:35.317333 28441 raft_consensus.cc:374] T b78c1e264d414455ab3f8e9996042a24 P a311dd73aa814f63b863f901521cdcea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:12:35.317415 28441 raft_consensus.cc:729] T b78c1e264d414455ab3f8e9996042a24 P a311dd73aa814f63b863f901521cdcea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a311dd73aa814f63b863f901521cdcea, State: Initialized, Role: FOLLOWER
I20260926 15:12:35.317996 28441 consensus_queue.cc:260] T b78c1e264d414455ab3f8e9996042a24 P a311dd73aa814f63b863f901521cdcea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e3b1a3f8e1343239a873c89195df725" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 44121 } } peers { permanent_uuid: "9fda66f1b152460e8e0cb199f028a5c0" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 38857 } } peers { permanent_uuid: "a311dd73aa814f63b863f901521cdcea" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 45901 } }
I20260926 15:12:35.320616 28441 ts_tablet_manager.cc:1428] T b78c1e264d414455ab3f8e9996042a24 P a311dd73aa814f63b863f901521cdcea: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260926 15:12:35.363554 28446 raft_consensus.cc:482] T b78c1e264d414455ab3f8e9996042a24 P a311dd73aa814f63b863f901521cdcea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 15:12:35.363953 28446 raft_consensus.cc:504] T b78c1e264d414455ab3f8e9996042a24 P a311dd73aa814f63b863f901521cdcea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e3b1a3f8e1343239a873c89195df725" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 44121 } } peers { permanent_uuid: "9fda66f1b152460e8e0cb199f028a5c0" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 38857 } } peers { permanent_uuid: "a311dd73aa814f63b863f901521cdcea" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 45901 } }
I20260926 15:12:35.365863 28446 leader_election.cc:288] T b78c1e264d414455ab3f8e9996042a24 P a311dd73aa814f63b863f901521cdcea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6e3b1a3f8e1343239a873c89195df725 (127.12.34.195:44121), 9fda66f1b152460e8e0cb199f028a5c0 (127.12.34.194:38857)
I20260926 15:12:35.373628 28433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:35.366321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.193:50467 (local address 127.12.34.195:44121)
0926 15:12:35.366553 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0926 15:12:35.366560 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:12:35.367207 (+ 647us) server_negotiation.cc:407] Connection header received
0926 15:12:35.367446 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:12:35.367450 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:12:35.367510 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:12:35.367609 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:12:35.368236 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:12:35.368990 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:12:35.371786 (+ 2796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:12:35.372544 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:12:35.372698 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:12:35.372787 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 15:12:35.373214 (+ 427us) server_negotiation.cc:299] Negotiation successful
0926 15:12:35.373355 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260926 15:12:35.373697 28435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:35.366418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.193:56649 (local address 127.12.34.194:38857)
0926 15:12:35.366623 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0926 15:12:35.366628 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:12:35.367326 (+ 698us) server_negotiation.cc:407] Connection header received
0926 15:12:35.367535 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:12:35.367546 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:12:35.367606 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:12:35.367703 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:12:35.368377 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:12:35.369202 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:12:35.371824 (+ 2622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:12:35.372568 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:12:35.372770 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:12:35.372840 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 15:12:35.373306 (+ 466us) server_negotiation.cc:299] Negotiation successful
0926 15:12:35.373431 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 15:12:35.373787 28447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:35.366186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.195:44121 (local address 127.12.34.193:50467)
0926 15:12:35.367071 (+ 885us) negotiation.cc:107] Waiting for socket to connect
0926 15:12:35.367100 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0926 15:12:35.367285 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:12:35.367636 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:12:35.367640 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:12:35.367666 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:12:35.368078 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:12:35.368087 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:12:35.369145 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:12:35.369153 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:12:35.371624 (+ 2471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:12:35.371640 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:12:35.372787 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:12:35.372793 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:12:35.372993 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:12:35.373017 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 15:12:35.373178 (+ 161us) client_negotiation.cc:240] Negotiation successful
0926 15:12:35.373305 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":734,"mutex_wait_us":96,"spinlock_wait_cycles":27520,"thread_start_us":461,"threads_started":1}
I20260926 15:12:35.373848 28448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:35.366418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.194:38857 (local address 127.12.34.193:56649)
0926 15:12:35.367216 (+ 798us) negotiation.cc:107] Waiting for socket to connect
0926 15:12:35.367230 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 15:12:35.367384 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:12:35.367724 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:12:35.367729 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:12:35.367833 (+ 104us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:12:35.368208 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:12:35.368216 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:12:35.369349 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:12:35.369356 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:12:35.371634 (+ 2278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:12:35.371643 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:12:35.372836 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:12:35.372843 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:12:35.373126 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:12:35.373160 (+ 34us) client_negotiation.cc:769] Sending connection context
0926 15:12:35.373280 (+ 120us) client_negotiation.cc:240] Negotiation successful
0926 15:12:35.373403 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":705,"mutex_wait_us":142,"spinlock_wait_cycles":30080,"thread_start_us":477,"threads_started":1}
I20260926 15:12:35.374749 28375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b78c1e264d414455ab3f8e9996042a24" candidate_uuid: "a311dd73aa814f63b863f901521cdcea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e3b1a3f8e1343239a873c89195df725" is_pre_election: true
I20260926 15:12:35.374794 28242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b78c1e264d414455ab3f8e9996042a24" candidate_uuid: "a311dd73aa814f63b863f901521cdcea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9fda66f1b152460e8e0cb199f028a5c0" is_pre_election: true
I20260926 15:12:35.375237 28375 raft_consensus.cc:2454] T b78c1e264d414455ab3f8e9996042a24 P 6e3b1a3f8e1343239a873c89195df725 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a311dd73aa814f63b863f901521cdcea in term 0.
I20260926 15:12:35.375253 28242 raft_consensus.cc:2454] T b78c1e264d414455ab3f8e9996042a24 P 9fda66f1b152460e8e0cb199f028a5c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a311dd73aa814f63b863f901521cdcea in term 0.
I20260926 15:12:35.375864 28044 leader_election.cc:302] T b78c1e264d414455ab3f8e9996042a24 P a311dd73aa814f63b863f901521cdcea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e3b1a3f8e1343239a873c89195df725, a311dd73aa814f63b863f901521cdcea; no voters:
I20260926 15:12:35.376224 28446 raft_consensus.cc:2781] T b78c1e264d414455ab3f8e9996042a24 P a311dd73aa814f63b863f901521cdcea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 15:12:35.376376 28446 raft_consensus.cc:482] T b78c1e264d414455ab3f8e9996042a24 P a311dd73aa814f63b863f901521cdcea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 15:12:35.376504 28446 raft_consensus.cc:3037] T b78c1e264d414455ab3f8e9996042a24 P a311dd73aa814f63b863f901521cdcea [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:12:35.378829 28446 raft_consensus.cc:504] T b78c1e264d414455ab3f8e9996042a24 P a311dd73aa814f63b863f901521cdcea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e3b1a3f8e1343239a873c89195df725" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 44121 } } peers { permanent_uuid: "9fda66f1b152460e8e0cb199f028a5c0" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 38857 } } peers { permanent_uuid: "a311dd73aa814f63b863f901521cdcea" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 45901 } }
I20260926 15:12:35.379442 28446 leader_election.cc:288] T b78c1e264d414455ab3f8e9996042a24 P a311dd73aa814f63b863f901521cdcea [CANDIDATE]: Term 1 election: Requested vote from peers 6e3b1a3f8e1343239a873c89195df725 (127.12.34.195:44121), 9fda66f1b152460e8e0cb199f028a5c0 (127.12.34.194:38857)
I20260926 15:12:35.379812 28375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b78c1e264d414455ab3f8e9996042a24" candidate_uuid: "a311dd73aa814f63b863f901521cdcea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e3b1a3f8e1343239a873c89195df725"
I20260926 15:12:35.379895 28242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b78c1e264d414455ab3f8e9996042a24" candidate_uuid: "a311dd73aa814f63b863f901521cdcea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9fda66f1b152460e8e0cb199f028a5c0"
I20260926 15:12:35.380034 28375 raft_consensus.cc:3037] T b78c1e264d414455ab3f8e9996042a24 P 6e3b1a3f8e1343239a873c89195df725 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:12:35.380136 28242 raft_consensus.cc:3037] T b78c1e264d414455ab3f8e9996042a24 P 9fda66f1b152460e8e0cb199f028a5c0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:12:35.382308 28375 raft_consensus.cc:2454] T b78c1e264d414455ab3f8e9996042a24 P 6e3b1a3f8e1343239a873c89195df725 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a311dd73aa814f63b863f901521cdcea in term 1.
I20260926 15:12:35.382531 28242 raft_consensus.cc:2454] T b78c1e264d414455ab3f8e9996042a24 P 9fda66f1b152460e8e0cb199f028a5c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a311dd73aa814f63b863f901521cdcea in term 1.
I20260926 15:12:35.382725 28044 leader_election.cc:302] T b78c1e264d414455ab3f8e9996042a24 P a311dd73aa814f63b863f901521cdcea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e3b1a3f8e1343239a873c89195df725, a311dd73aa814f63b863f901521cdcea; no voters:
I20260926 15:12:35.382997 28446 raft_consensus.cc:2781] T b78c1e264d414455ab3f8e9996042a24 P a311dd73aa814f63b863f901521cdcea [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:12:35.383469 28446 raft_consensus.cc:686] T b78c1e264d414455ab3f8e9996042a24 P a311dd73aa814f63b863f901521cdcea [term 1 LEADER]: Becoming Leader. State: Replica: a311dd73aa814f63b863f901521cdcea, State: Running, Role: LEADER
I20260926 15:12:35.384012 28446 consensus_queue.cc:237] T b78c1e264d414455ab3f8e9996042a24 P a311dd73aa814f63b863f901521cdcea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e3b1a3f8e1343239a873c89195df725" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 44121 } } peers { permanent_uuid: "9fda66f1b152460e8e0cb199f028a5c0" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 38857 } } peers { permanent_uuid: "a311dd73aa814f63b863f901521cdcea" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 45901 } }
I20260926 15:12:35.388381 27964 catalog_manager.cc:5495] T b78c1e264d414455ab3f8e9996042a24 P a311dd73aa814f63b863f901521cdcea reported cstate change: term changed from 0 to 1, leader changed from <none> to a311dd73aa814f63b863f901521cdcea (127.12.34.193). New cstate: current_term: 1 leader_uuid: "a311dd73aa814f63b863f901521cdcea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e3b1a3f8e1343239a873c89195df725" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 44121 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9fda66f1b152460e8e0cb199f028a5c0" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 38857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a311dd73aa814f63b863f901521cdcea" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 45901 } health_report { overall_health: HEALTHY } } }
I20260926 15:12:35.405012 28437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:35.399758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43268 (local address 127.12.34.193:45901)
0926 15:12:35.399957 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0926 15:12:35.399964 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:12:35.400116 (+ 152us) server_negotiation.cc:407] Connection header received
0926 15:12:35.400271 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:12:35.400275 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:12:35.400353 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:12:35.400477 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 15:12:35.400792 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:12:35.401544 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:12:35.402922 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:12:35.403352 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:12:35.403450 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:12:35.403883 (+ 433us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 15:12:35.404469 (+ 586us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 15:12:35.404580 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0926 15:12:35.404710 (+ 130us) server_negotiation.cc:299] Negotiation successful
0926 15:12:35.404769 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 15:12:35.412472 28242 raft_consensus.cc:1261] T b78c1e264d414455ab3f8e9996042a24 P 9fda66f1b152460e8e0cb199f028a5c0 [term 1 FOLLOWER]: Refusing update from remote peer a311dd73aa814f63b863f901521cdcea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:12:35.412528 28375 raft_consensus.cc:1261] T b78c1e264d414455ab3f8e9996042a24 P 6e3b1a3f8e1343239a873c89195df725 [term 1 FOLLOWER]: Refusing update from remote peer a311dd73aa814f63b863f901521cdcea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:12:35.413407 28449 consensus_queue.cc:1035] T b78c1e264d414455ab3f8e9996042a24 P a311dd73aa814f63b863f901521cdcea [LEADER]: Connected to new peer: Peer: permanent_uuid: "9fda66f1b152460e8e0cb199f028a5c0" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 38857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:12:35.413735 28446 consensus_queue.cc:1035] T b78c1e264d414455ab3f8e9996042a24 P a311dd73aa814f63b863f901521cdcea [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e3b1a3f8e1343239a873c89195df725" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 44121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:12:35.439510 27964 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49888:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 15:12:35.439728 27964 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49888:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 15:12:35.443431 27964 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7756b74c7007403c85a266f55693c0bc: Sending DeleteTablet for 3 replicas of tablet b78c1e264d414455ab3f8e9996042a24
I20260926 15:12:35.444442 28355 tablet_service.cc:1514] Processing DeleteTablet for tablet b78c1e264d414455ab3f8e9996042a24 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:12:35 UTC) from {username='kudu', principal='kudu/127.12.34.254@KRBTEST.COM'} at 127.0.0.1:40196
I20260926 15:12:35.444742 12427 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 28025
I20260926 15:12:35.444959 28222 tablet_service.cc:1514] Processing DeleteTablet for tablet b78c1e264d414455ab3f8e9996042a24 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:12:35 UTC) from {username='kudu', principal='kudu/127.12.34.254@KRBTEST.COM'} at 127.0.0.1:51178
I20260926 15:12:35.445454 28088 tablet_service.cc:1514] Processing DeleteTablet for tablet b78c1e264d414455ab3f8e9996042a24 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:12:35 UTC) from {username='kudu', principal='kudu/127.12.34.254@KRBTEST.COM'} at 127.0.0.1:43264
I20260926 15:12:35.445560 28462 tablet_replica.cc:330] T b78c1e264d414455ab3f8e9996042a24 P 6e3b1a3f8e1343239a873c89195df725: stopping tablet replica
I20260926 15:12:35.445930 28462 raft_consensus.cc:2229] T b78c1e264d414455ab3f8e9996042a24 P 6e3b1a3f8e1343239a873c89195df725 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 15:12:35.446466 28462 raft_consensus.cc:2258] T b78c1e264d414455ab3f8e9996042a24 P 6e3b1a3f8e1343239a873c89195df725 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:12:35.448889 28462 ts_tablet_manager.cc:1905] T b78c1e264d414455ab3f8e9996042a24 P 6e3b1a3f8e1343239a873c89195df725: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:12:35.457707 28463 tablet_replica.cc:330] T b78c1e264d414455ab3f8e9996042a24 P 9fda66f1b152460e8e0cb199f028a5c0: stopping tablet replica
I20260926 15:12:35.458115 28463 raft_consensus.cc:2229] T b78c1e264d414455ab3f8e9996042a24 P 9fda66f1b152460e8e0cb199f028a5c0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 15:12:35.458571 28463 raft_consensus.cc:2258] T b78c1e264d414455ab3f8e9996042a24 P 9fda66f1b152460e8e0cb199f028a5c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:12:35.458838 28462 ts_tablet_manager.cc:1918] T b78c1e264d414455ab3f8e9996042a24 P 6e3b1a3f8e1343239a873c89195df725: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 15:12:35.458998 28462 log.cc:1196] T b78c1e264d414455ab3f8e9996042a24 P 6e3b1a3f8e1343239a873c89195df725: Deleting WAL directory at /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/ts-2/wal/wals/b78c1e264d414455ab3f8e9996042a24
I20260926 15:12:35.459313 28465 tablet_replica.cc:330] T b78c1e264d414455ab3f8e9996042a24 P a311dd73aa814f63b863f901521cdcea: stopping tablet replica
I20260926 15:12:35.459532 28462 ts_tablet_manager.cc:1939] T b78c1e264d414455ab3f8e9996042a24 P 6e3b1a3f8e1343239a873c89195df725: Deleting consensus metadata
I20260926 15:12:35.459667 28465 raft_consensus.cc:2229] T b78c1e264d414455ab3f8e9996042a24 P a311dd73aa814f63b863f901521cdcea [term 1 LEADER]: Raft consensus shutting down.
I20260926 15:12:35.460405 28465 raft_consensus.cc:2258] T b78c1e264d414455ab3f8e9996042a24 P a311dd73aa814f63b863f901521cdcea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:12:35.461289 27951 catalog_manager.cc:4841] TS 6e3b1a3f8e1343239a873c89195df725 (127.12.34.195:44121): tablet b78c1e264d414455ab3f8e9996042a24 (table test-table [id=0c1066636dff487283a72186f69cb0b0]) successfully deleted
I20260926 15:12:35.463012 28463 ts_tablet_manager.cc:1905] T b78c1e264d414455ab3f8e9996042a24 P 9fda66f1b152460e8e0cb199f028a5c0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:12:35.463303 28465 ts_tablet_manager.cc:1905] T b78c1e264d414455ab3f8e9996042a24 P a311dd73aa814f63b863f901521cdcea: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 26 15:12:35 dist-test-slave-865v krb5kdc[27919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435553, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.34.193@KRBTEST.COM
I20260926 15:12:35.469894 28463 ts_tablet_manager.cc:1918] T b78c1e264d414455ab3f8e9996042a24 P 9fda66f1b152460e8e0cb199f028a5c0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 15:12:35.470052 28463 log.cc:1196] T b78c1e264d414455ab3f8e9996042a24 P 9fda66f1b152460e8e0cb199f028a5c0: Deleting WAL directory at /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/ts-1/wal/wals/b78c1e264d414455ab3f8e9996042a24
I20260926 15:12:35.470587 28463 ts_tablet_manager.cc:1939] T b78c1e264d414455ab3f8e9996042a24 P 9fda66f1b152460e8e0cb199f028a5c0: Deleting consensus metadata
I20260926 15:12:35.470608 28465 ts_tablet_manager.cc:1918] T b78c1e264d414455ab3f8e9996042a24 P a311dd73aa814f63b863f901521cdcea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 15:12:35.470764 28465 log.cc:1196] T b78c1e264d414455ab3f8e9996042a24 P a311dd73aa814f63b863f901521cdcea: Deleting WAL directory at /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/ts-0/wal/wals/b78c1e264d414455ab3f8e9996042a24
I20260926 15:12:35.472198 27950 catalog_manager.cc:4841] TS 9fda66f1b152460e8e0cb199f028a5c0 (127.12.34.194:38857): tablet b78c1e264d414455ab3f8e9996042a24 (table test-table [id=0c1066636dff487283a72186f69cb0b0]) successfully deleted
I20260926 15:12:35.472577 28465 ts_tablet_manager.cc:1939] T b78c1e264d414455ab3f8e9996042a24 P a311dd73aa814f63b863f901521cdcea: Deleting consensus metadata
I20260926 15:12:35.474064 27950 catalog_manager.cc:4841] TS a311dd73aa814f63b863f901521cdcea (127.12.34.193:45901): tablet b78c1e264d414455ab3f8e9996042a24 (table test-table [id=0c1066636dff487283a72186f69cb0b0]) successfully deleted
I20260926 15:12:35.477581 28437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:35.445860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43278 (local address 127.12.34.193:45901)
0926 15:12:35.446064 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0926 15:12:35.446070 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:12:35.456770 (+ 10700us) server_negotiation.cc:407] Connection header received
0926 15:12:35.457221 (+ 451us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:12:35.457229 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:12:35.457314 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:12:35.457390 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:12:35.459938 (+ 2548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:12:35.461163 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:12:35.463883 (+ 2720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:12:35.464592 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:12:35.464695 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:12:35.470653 (+ 5958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:12:35.470667 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:12:35.470684 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:12:35.470738 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:12:35.474828 (+ 4090us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:12:35.475496 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:12:35.475505 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:12:35.475509 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:12:35.475691 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:12:35.476050 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:12:35.476055 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:12:35.476059 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:12:35.476459 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:12:35.476757 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0926 15:12:35.476967 (+ 210us) server_negotiation.cc:299] Negotiation successful
0926 15:12:35.477213 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 15:12:35.478228 28148 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:12:35.574172 12427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8uoJGv/build/asan/bin/kudu with pid 28025
I20260926 15:12:35.593053 12427 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 28159
Sep 26 15:12:35 dist-test-slave-865v krb5kdc[27919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435553, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.34.194@KRBTEST.COM
I20260926 15:12:35.608687 28435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:35.593828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51188 (local address 127.12.34.194:38857)
0926 15:12:35.593993 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0926 15:12:35.593998 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:12:35.594014 (+ 16us) server_negotiation.cc:407] Connection header received
0926 15:12:35.594170 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:12:35.594174 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:12:35.594244 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:12:35.594327 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:12:35.595516 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:12:35.596231 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:12:35.597815 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:12:35.598190 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:12:35.598281 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:12:35.602902 (+ 4621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:12:35.602915 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:12:35.602930 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:12:35.602962 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:12:35.605916 (+ 2954us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:12:35.606607 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:12:35.606614 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:12:35.606616 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:12:35.606731 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:12:35.607120 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:12:35.607126 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:12:35.607129 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:12:35.607513 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:12:35.607811 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0926 15:12:35.608137 (+ 326us) server_negotiation.cc:299] Negotiation successful
0926 15:12:35.608366 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 15:12:35.609351 28282 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:12:35.699982 12427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8uoJGv/build/asan/bin/kudu with pid 28159
I20260926 15:12:35.716629 12427 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 28292
Sep 26 15:12:35 dist-test-slave-865v krb5kdc[27919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435553, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.34.195@KRBTEST.COM
I20260926 15:12:35.731912 28433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:35.717448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40210 (local address 127.12.34.195:44121)
0926 15:12:35.717622 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0926 15:12:35.717627 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:12:35.717647 (+ 20us) server_negotiation.cc:407] Connection header received
0926 15:12:35.717782 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:12:35.717786 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:12:35.717856 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:12:35.717943 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:12:35.719340 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:12:35.720117 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:12:35.721613 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:12:35.722053 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:12:35.722155 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:12:35.726398 (+ 4243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:12:35.726412 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:12:35.726428 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:12:35.726464 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:12:35.729287 (+ 2823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:12:35.729930 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:12:35.729945 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:12:35.729948 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:12:35.730086 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:12:35.730446 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:12:35.730450 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:12:35.730452 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:12:35.730822 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:12:35.731040 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0926 15:12:35.731383 (+ 343us) server_negotiation.cc:299] Negotiation successful
0926 15:12:35.731552 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 15:12:35.732648 28415 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:12:35.825300 12427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8uoJGv/build/asan/bin/kudu with pid 28292
I20260926 15:12:35.842504 12427 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 27932
I20260926 15:12:35.843657 27994 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:12:35.952734 12427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8uoJGv/build/asan/bin/kudu with pid 27932
I20260926 15:12:55.974385 12427 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8uoJGv/build/asan/bin/kudu
/tmp/dist-test-task8uoJGv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.34.254:45601
--webserver_interface=127.12.34.254
--webserver_port=43841
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.254.keytab
--principal=kudu/127.12.34.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.34.212:40893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.34.254:45601
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:12:56.298360 28470 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:12:56.298669 28470 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:12:56.298837 28470 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:12:56.306011 28470 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 15:12:56.306133 28470 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 15:12:56.306229 28470 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 15:12:56.306324 28470 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:12:56.306427 28470 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:12:56.306531 28470 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:12:56.306602 28470 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:12:56.306682 28470 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:12:56.316319 28470 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.34.212:40893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.12.34.254:45601
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.254.keytab
--principal=kudu/127.12.34.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.34.254:45601
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.34.254
--webserver_port=43841
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:12:56.317577 28470 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:12:56.318856 28470 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:12:56.343035 28477 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:12:56.343098 28476 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:12:56.342993 28479 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:12:56.343683 28470 server_base.cc:1034] running on GCE node
I20260926 15:12:56.344592 28470 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:12:56.346289 28470 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:12:56.347527 28470 hybrid_clock.cc:648] HybridClock initialized: now 1790435576347475 us; error 28 us; skew 500 ppm
I20260926 15:12:56.347940 28470 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:12:56 dist-test-slave-865v krb5kdc[27919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435576, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:12:56.351922 28470 init.cc:377] Logged in from keytab as kudu/127.12.34.254@KRBTEST.COM (short username kudu)
I20260926 15:12:56.353751 28470 webserver.cc:460] Webserver started at http://127.12.34.254:43841/ using document root <none> and password file <none>
I20260926 15:12:56.354593 28470 fs_manager.cc:359] Metadata directory not provided
I20260926 15:12:56.354758 28470 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:12:56.361382 28470 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.003s
I20260926 15:12:56.364351 28486 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:12:56.365899 28470 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 15:12:56.366114 28470 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/master-0/data,/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/master-0/wal
uuid: "7756b74c7007403c85a266f55693c0bc"
format_stamp: "Formatted at 2026-09-26 15:12:33 on dist-test-slave-865v"
server_key: "3adac6befea70a01de26d053623217b3"
server_key_iv: "eb96e0558a0965c6ed14ad592c0b6369"
server_key_version: "encryptionkey@0"
I20260926 15:12:56.367302 28470 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:12:56.394654 28470 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:12:56.399216 28470 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:12:56.399544 28470 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:12:56.419924 28470 rpc_server.cc:307] RPC server started. Bound to: 127.12.34.254:45601
I20260926 15:12:56.419963 28537 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.34.254:45601 every 8 connection(s)
I20260926 15:12:56.421574 28470 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
I20260926 15:12:56.426990 12427 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8uoJGv/build/asan/bin/kudu as pid 28470
I20260926 15:12:56.427888 12427 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8uoJGv/build/asan/bin/kudu
/tmp/dist-test-task8uoJGv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.34.193:45901
--local_ip_for_outbound_sockets=127.12.34.193
--tserver_master_addrs=127.12.34.254:45601
--webserver_port=36813
--webserver_interface=127.12.34.193
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.193.keytab
--principal=kudu/127.12.34.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.34.212:40893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 15:12:56.428859 28538 sys_catalog.cc:263] Verifying existing consensus state
I20260926 15:12:56.433460 28538 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7756b74c7007403c85a266f55693c0bc: Bootstrap starting.
I20260926 15:12:56.452960 28538 log.cc:824] T 00000000000000000000000000000000 P 7756b74c7007403c85a266f55693c0bc: Log is configured to *not* fsync() on all Append() calls
I20260926 15:12:56.464484 28538 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7756b74c7007403c85a266f55693c0bc: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 15:12:56.465036 28538 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7756b74c7007403c85a266f55693c0bc: Bootstrap complete.
I20260926 15:12:56.474781 28538 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7756b74c7007403c85a266f55693c0bc [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7756b74c7007403c85a266f55693c0bc" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 45601 } }
I20260926 15:12:56.475622 28538 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7756b74c7007403c85a266f55693c0bc [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7756b74c7007403c85a266f55693c0bc, State: Initialized, Role: FOLLOWER
I20260926 15:12:56.476188 28538 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7756b74c7007403c85a266f55693c0bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7756b74c7007403c85a266f55693c0bc" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 45601 } }
I20260926 15:12:56.476440 28538 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7756b74c7007403c85a266f55693c0bc [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 15:12:56.476572 28538 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7756b74c7007403c85a266f55693c0bc [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 15:12:56.476751 28538 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7756b74c7007403c85a266f55693c0bc [term 1 FOLLOWER]: Advancing to term 2
I20260926 15:12:56.480567 28538 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7756b74c7007403c85a266f55693c0bc [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7756b74c7007403c85a266f55693c0bc" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 45601 } }
I20260926 15:12:56.481185 28538 leader_election.cc:302] T 00000000000000000000000000000000 P 7756b74c7007403c85a266f55693c0bc [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7756b74c7007403c85a266f55693c0bc; no voters:
I20260926 15:12:56.481835 28538 leader_election.cc:288] T 00000000000000000000000000000000 P 7756b74c7007403c85a266f55693c0bc [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 15:12:56.481959 28542 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7756b74c7007403c85a266f55693c0bc [term 2 FOLLOWER]: Leader election won for term 2
I20260926 15:12:56.483181 28542 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7756b74c7007403c85a266f55693c0bc [term 2 LEADER]: Becoming Leader. State: Replica: 7756b74c7007403c85a266f55693c0bc, State: Running, Role: LEADER
I20260926 15:12:56.483776 28542 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7756b74c7007403c85a266f55693c0bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7756b74c7007403c85a266f55693c0bc" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 45601 } }
I20260926 15:12:56.484169 28538 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7756b74c7007403c85a266f55693c0bc [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:12:56.486256 28544 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7756b74c7007403c85a266f55693c0bc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7756b74c7007403c85a266f55693c0bc. Latest consensus state: current_term: 2 leader_uuid: "7756b74c7007403c85a266f55693c0bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7756b74c7007403c85a266f55693c0bc" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 45601 } } }
I20260926 15:12:56.486709 28544 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7756b74c7007403c85a266f55693c0bc [sys.catalog]: This master's current role is: LEADER
I20260926 15:12:56.486629 28543 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7756b74c7007403c85a266f55693c0bc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "7756b74c7007403c85a266f55693c0bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7756b74c7007403c85a266f55693c0bc" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 45601 } } }
I20260926 15:12:56.487028 28543 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7756b74c7007403c85a266f55693c0bc [sys.catalog]: This master's current role is: LEADER
I20260926 15:12:56.491987 28547 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:12:56.503546 28547 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 15:12:56.504988 28547 catalog_manager.cc:1259] Loaded cluster ID: dd2c37d372564116a5191e490b674674
I20260926 15:12:56.505151 28547 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 15:12:56.509161 28547 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:12:56.510686 28547 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 7756b74c7007403c85a266f55693c0bc: Loaded TSK: 0
I20260926 15:12:56.545050 28547 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7756b74c7007403c85a266f55693c0bc: Generated new TSK 1
I20260926 15:12:56.545487 28547 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 15:12:56.769198 28540 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:12:56.769526 28540 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:12:56.769688 28540 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:12:56.777334 28540 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:12:56.777490 28540 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:12:56.777662 28540 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.34.193
I20260926 15:12:56.787708 28540 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.34.212:40893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.193.keytab
--principal=kudu/127.12.34.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.34.193:45901
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.34.193
--webserver_port=36813
--tserver_master_addrs=127.12.34.254:45601
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.34.193
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:12:56.789494 28540 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:12:56.791551 28540 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:12:56.806684 28567 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:12:56.807209 28569 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:12:56.806663 28566 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:12:56.808609 28540 server_base.cc:1034] running on GCE node
I20260926 15:12:56.809238 28540 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:12:56.810190 28540 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:12:56.811513 28540 hybrid_clock.cc:648] HybridClock initialized: now 1790435576811451 us; error 49 us; skew 500 ppm
I20260926 15:12:56.811857 28540 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:12:56 dist-test-slave-865v krb5kdc[27919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435576, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:12:56.815373 28540 init.cc:377] Logged in from keytab as kudu/127.12.34.193@KRBTEST.COM (short username kudu)
I20260926 15:12:56.817391 28540 webserver.cc:460] Webserver started at http://127.12.34.193:36813/ using document root <none> and password file <none>
I20260926 15:12:56.818285 28540 fs_manager.cc:359] Metadata directory not provided
I20260926 15:12:56.818445 28540 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:12:56.825394 28540 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.003s
I20260926 15:12:56.828687 28576 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:12:56.830323 28540 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 15:12:56.830487 28540 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/ts-0/data,/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/ts-0/wal
uuid: "a311dd73aa814f63b863f901521cdcea"
format_stamp: "Formatted at 2026-09-26 15:12:34 on dist-test-slave-865v"
server_key: "c274070bb383f9b2ce7d764574679349"
server_key_iv: "6e617dfd8ed626a53b3b93bba0428b34"
server_key_version: "encryptionkey@0"
I20260926 15:12:56.831802 28540 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:12:56.871775 28540 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:12:56.877300 28540 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:12:56.877580 28540 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:12:56.878702 28540 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:12:56.880303 28540 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:12:56.880409 28540 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:12:56.880509 28540 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:12:56.880584 28540 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:12:56.915306 28540 rpc_server.cc:307] RPC server started. Bound to: 127.12.34.193:45901
I20260926 15:12:56.915426 28688 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.34.193:45901 every 8 connection(s)
I20260926 15:12:56.916726 28540 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/ts-0/data/info.pb
I20260926 15:12:56.922703 12427 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8uoJGv/build/asan/bin/kudu as pid 28540
I20260926 15:12:56.923573 12427 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8uoJGv/build/asan/bin/kudu
/tmp/dist-test-task8uoJGv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.34.194:38857
--local_ip_for_outbound_sockets=127.12.34.194
--tserver_master_addrs=127.12.34.254:45601
--webserver_port=36799
--webserver_interface=127.12.34.194
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.194.keytab
--principal=kudu/127.12.34.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.34.212:40893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 15:12:56 dist-test-slave-865v krb5kdc[27919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435576, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.193@KRBTEST.COM for kudu/127.12.34.254@KRBTEST.COM
I20260926 15:12:56.949731 28691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:56.919821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.193:43535 (local address 127.12.34.254:45601)
0926 15:12:56.920401 (+ 580us) server_negotiation.cc:206] Beginning negotiation
0926 15:12:56.920413 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 15:12:56.921253 (+ 840us) server_negotiation.cc:407] Connection header received
0926 15:12:56.924228 (+ 2975us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:12:56.924240 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:12:56.924553 (+ 313us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:12:56.924842 (+ 289us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:12:56.927056 (+ 2214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:12:56.928794 (+ 1738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:12:56.931012 (+ 2218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:12:56.931951 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:12:56.932147 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:12:56.941253 (+ 9106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:12:56.941261 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:12:56.941275 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:12:56.941314 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:12:56.944461 (+ 3147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:12:56.945685 (+ 1224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:12:56.945692 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:12:56.945697 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:12:56.945895 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:12:56.946487 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:12:56.946492 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:12:56.946494 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:12:56.947033 (+ 539us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:12:56.947193 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0926 15:12:56.948315 (+ 1122us) server_negotiation.cc:299] Negotiation successful
0926 15:12:56.948656 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399,"thread_start_us":245,"threads_started":1}
I20260926 15:12:56.950356 28692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:56.920133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:45601 (local address 127.12.34.193:43535)
0926 15:12:56.921026 (+ 893us) negotiation.cc:107] Waiting for socket to connect
0926 15:12:56.921105 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0926 15:12:56.923654 (+ 2549us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:12:56.925073 (+ 1419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:12:56.925142 (+ 69us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:12:56.925982 (+ 840us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:12:56.926827 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:12:56.926856 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:12:56.928999 (+ 2143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:12:56.929006 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:12:56.930845 (+ 1839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:12:56.930862 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:12:56.932190 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:12:56.932197 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:12:56.932461 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:12:56.938228 (+ 5767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:12:56.938252 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:12:56.940863 (+ 2611us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:12:56.944638 (+ 3775us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:12:56.944649 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:12:56.944668 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:12:56.945517 (+ 849us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:12:56.946028 (+ 511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:12:56.946033 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:12:56.946039 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:12:56.946327 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:12:56.947191 (+ 864us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:12:56.947204 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:12:56.947720 (+ 516us) client_negotiation.cc:769] Sending connection context
0926 15:12:56.948110 (+ 390us) client_negotiation.cc:240] Negotiation successful
0926 15:12:56.948796 (+ 686us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":601,"thread_start_us":391,"threads_started":1}
I20260926 15:12:56.953532 28689 heartbeater.cc:347] Connected to a master server at 127.12.34.254:45601
I20260926 15:12:56.953845 28689 heartbeater.cc:464] Registering TS with master...
I20260926 15:12:56.954632 28689 heartbeater.cc:511] Master 127.12.34.254:45601 requested a full tablet report, sending...
I20260926 15:12:56.957052 28503 ts_manager.cc:194] Registered new tserver with Master: a311dd73aa814f63b863f901521cdcea (127.12.34.193:45901)
I20260926 15:12:56.959324 28503 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.34.193@KRBTEST.COM'} at 127.12.34.193:43535
W20260926 15:12:57.264356 28693 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:12:57.264642 28693 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:12:57.264742 28693 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:12:57.272264 28693 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:12:57.272372 28693 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:12:57.272487 28693 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.34.194
I20260926 15:12:57.282754 28693 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.34.212:40893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.194.keytab
--principal=kudu/127.12.34.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.34.194:38857
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.34.194
--webserver_port=36799
--tserver_master_addrs=127.12.34.254:45601
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.34.194
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:12:57.284040 28693 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:12:57.285422 28693 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:12:57.298225 28701 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:12:57.298264 28699 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:12:57.298444 28698 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:12:57.299196 28693 server_base.cc:1034] running on GCE node
I20260926 15:12:57.299940 28693 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:12:57.301038 28693 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:12:57.302320 28693 hybrid_clock.cc:648] HybridClock initialized: now 1790435577302258 us; error 56 us; skew 500 ppm
I20260926 15:12:57.302635 28693 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:12:57 dist-test-slave-865v krb5kdc[27919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435577, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:12:57.305945 28693 init.cc:377] Logged in from keytab as kudu/127.12.34.194@KRBTEST.COM (short username kudu)
I20260926 15:12:57.307367 28693 webserver.cc:460] Webserver started at http://127.12.34.194:36799/ using document root <none> and password file <none>
I20260926 15:12:57.308020 28693 fs_manager.cc:359] Metadata directory not provided
I20260926 15:12:57.308173 28693 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:12:57.313936 28693 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 15:12:57.316560 28708 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:12:57.317899 28693 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 15:12:57.318077 28693 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/ts-1/data,/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/ts-1/wal
uuid: "9fda66f1b152460e8e0cb199f028a5c0"
format_stamp: "Formatted at 2026-09-26 15:12:34 on dist-test-slave-865v"
server_key: "fd39586b7675ea818ec3df58e8cc92a9"
server_key_iv: "d6367cc0dc316147b92055a8d8fed163"
server_key_version: "encryptionkey@0"
I20260926 15:12:57.319036 28693 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:12:57.350029 28693 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:12:57.353932 28693 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:12:57.354202 28693 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:12:57.355177 28693 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:12:57.356485 28693 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:12:57.356611 28693 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:12:57.356745 28693 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:12:57.356840 28693 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:12:57.391148 28693 rpc_server.cc:307] RPC server started. Bound to: 127.12.34.194:38857
I20260926 15:12:57.391232 28820 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.34.194:38857 every 8 connection(s)
I20260926 15:12:57.392562 28693 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/ts-1/data/info.pb
I20260926 15:12:57.400527 12427 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8uoJGv/build/asan/bin/kudu as pid 28693
I20260926 15:12:57.401171 12427 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8uoJGv/build/asan/bin/kudu
/tmp/dist-test-task8uoJGv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.34.195:44121
--local_ip_for_outbound_sockets=127.12.34.195
--tserver_master_addrs=127.12.34.254:45601
--webserver_port=33713
--webserver_interface=127.12.34.195
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.195.keytab
--principal=kudu/127.12.34.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.34.212:40893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 15:12:57 dist-test-slave-865v krb5kdc[27919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435577, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.194@KRBTEST.COM for kudu/127.12.34.254@KRBTEST.COM
I20260926 15:12:57.412410 28691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:57.395001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.194:52451 (local address 127.12.34.254:45601)
0926 15:12:57.395172 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0926 15:12:57.395177 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:12:57.396289 (+ 1112us) server_negotiation.cc:407] Connection header received
0926 15:12:57.398147 (+ 1858us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:12:57.398153 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:12:57.398245 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:12:57.398355 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:12:57.399729 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:12:57.400445 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:12:57.402100 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:12:57.402505 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:12:57.402635 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:12:57.406268 (+ 3633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:12:57.406277 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:12:57.406285 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:12:57.406327 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:12:57.409320 (+ 2993us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:12:57.410181 (+ 861us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:12:57.410185 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:12:57.410187 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:12:57.410298 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:12:57.410777 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:12:57.410781 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:12:57.410783 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:12:57.411046 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:12:57.411165 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0926 15:12:57.411996 (+ 831us) server_negotiation.cc:299] Negotiation successful
0926 15:12:57.412161 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 15:12:57.413812 28823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:57.395468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:45601 (local address 127.12.34.194:52451)
0926 15:12:57.396131 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0926 15:12:57.396170 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 15:12:57.397891 (+ 1721us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:12:57.398581 (+ 690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:12:57.398588 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:12:57.399056 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:12:57.399564 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:12:57.399578 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:12:57.400596 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:12:57.400603 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:12:57.401959 (+ 1356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:12:57.401972 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:12:57.402715 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:12:57.402722 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:12:57.402936 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:12:57.403601 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:12:57.403621 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:12:57.406028 (+ 2407us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:12:57.409446 (+ 3418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:12:57.409456 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:12:57.409474 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:12:57.410068 (+ 594us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:12:57.410409 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:12:57.410416 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:12:57.410423 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:12:57.410652 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:12:57.411175 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:12:57.411185 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:12:57.411618 (+ 433us) client_negotiation.cc:769] Sending connection context
0926 15:12:57.411897 (+ 279us) client_negotiation.cc:240] Negotiation successful
0926 15:12:57.412444 (+ 547us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":281,"threads_started":1}
I20260926 15:12:57.415239 28821 heartbeater.cc:347] Connected to a master server at 127.12.34.254:45601
I20260926 15:12:57.415555 28821 heartbeater.cc:464] Registering TS with master...
I20260926 15:12:57.416342 28821 heartbeater.cc:511] Master 127.12.34.254:45601 requested a full tablet report, sending...
I20260926 15:12:57.417827 28503 ts_manager.cc:194] Registered new tserver with Master: 9fda66f1b152460e8e0cb199f028a5c0 (127.12.34.194:38857)
I20260926 15:12:57.418706 28503 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.34.194@KRBTEST.COM'} at 127.12.34.194:52451
W20260926 15:12:57.726529 28824 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:12:57.726805 28824 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:12:57.726949 28824 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:12:57.734123 28824 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:12:57.734265 28824 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:12:57.734412 28824 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.34.195
I20260926 15:12:57.744161 28824 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.34.212:40893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.195.keytab
--principal=kudu/127.12.34.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.34.195:44121
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.34.195
--webserver_port=33713
--tserver_master_addrs=127.12.34.254:45601
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.34.195
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:12:57.745414 28824 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:12:57.746587 28824 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:12:57.759123 28829 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:12:57.759122 28830 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:12:57.759469 28832 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:12:57.760341 28824 server_base.cc:1034] running on GCE node
I20260926 15:12:57.760941 28824 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:12:57.761871 28824 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:12:57.763110 28824 hybrid_clock.cc:648] HybridClock initialized: now 1790435577763052 us; error 46 us; skew 500 ppm
I20260926 15:12:57.763427 28824 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:12:57 dist-test-slave-865v krb5kdc[27919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435577, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:12:57.766507 28824 init.cc:377] Logged in from keytab as kudu/127.12.34.195@KRBTEST.COM (short username kudu)
I20260926 15:12:57.767954 28824 webserver.cc:460] Webserver started at http://127.12.34.195:33713/ using document root <none> and password file <none>
I20260926 15:12:57.768625 28824 fs_manager.cc:359] Metadata directory not provided
I20260926 15:12:57.768783 28824 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:12:57.774523 28824 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 15:12:57.777325 28839 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:12:57.778620 28824 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 15:12:57.778890 28824 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/ts-2/data,/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/ts-2/wal
uuid: "6e3b1a3f8e1343239a873c89195df725"
format_stamp: "Formatted at 2026-09-26 15:12:35 on dist-test-slave-865v"
server_key: "5f8da085443f5841cf5692d98bb00261"
server_key_iv: "d68a194c90806156ba2e288e7d369e21"
server_key_version: "encryptionkey@0"
I20260926 15:12:57.779920 28824 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:12:57.799616 28824 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:12:57.803577 28824 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:12:57.803850 28824 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:12:57.804786 28824 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:12:57.806121 28824 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:12:57.806248 28824 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:12:57.806376 28824 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:12:57.806463 28824 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:12:57.839792 28824 rpc_server.cc:307] RPC server started. Bound to: 127.12.34.195:44121
I20260926 15:12:57.839855 28951 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.34.195:44121 every 8 connection(s)
I20260926 15:12:57.841213 28824 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/ts-2/data/info.pb
I20260926 15:12:57.846418 12427 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8uoJGv/build/asan/bin/kudu as pid 28824
Sep 26 15:12:57 dist-test-slave-865v krb5kdc[27919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435577, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.195@KRBTEST.COM for kudu/127.12.34.254@KRBTEST.COM
I20260926 15:12:57.861415 28956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:57.847444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43892 (local address 127.12.34.254:45601)
0926 15:12:57.848266 (+ 822us) server_negotiation.cc:206] Beginning negotiation
0926 15:12:57.848273 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:12:57.848294 (+ 21us) server_negotiation.cc:407] Connection header received
0926 15:12:57.848350 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:12:57.848353 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:12:57.848411 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:12:57.848531 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:12:57.850120 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:12:57.850877 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:12:57.852151 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:12:57.852638 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:12:57.852849 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:12:57.855680 (+ 2831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:12:57.855685 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:12:57.855688 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:12:57.855717 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:12:57.858582 (+ 2865us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:12:57.859253 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:12:57.859259 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:12:57.859269 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:12:57.859390 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:12:57.859764 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:12:57.859769 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:12:57.859772 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:12:57.860038 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:12:57.860294 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0926 15:12:57.860610 (+ 316us) server_negotiation.cc:299] Negotiation successful
0926 15:12:57.860817 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":42,"server-negotiator.queue_time_us":707,"thread_start_us":585,"threads_started":1}
I20260926 15:12:57.863452 28691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:57.843883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.195:53649 (local address 127.12.34.254:45601)
0926 15:12:57.844085 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0926 15:12:57.844091 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:12:57.845158 (+ 1067us) server_negotiation.cc:407] Connection header received
0926 15:12:57.847798 (+ 2640us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:12:57.847808 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:12:57.847927 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:12:57.848034 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:12:57.849826 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:12:57.850676 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:12:57.852322 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:12:57.852844 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:12:57.852997 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:12:57.856866 (+ 3869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:12:57.856871 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:12:57.856875 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:12:57.856918 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:12:57.859976 (+ 3058us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:12:57.860708 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:12:57.860713 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:12:57.860725 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:12:57.860841 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:12:57.861390 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:12:57.861407 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:12:57.861409 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:12:57.861768 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:12:57.861946 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0926 15:12:57.862831 (+ 885us) server_negotiation.cc:299] Negotiation successful
0926 15:12:57.863048 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":42,"server-negotiator.queue_time_us":66}
I20260926 15:12:57.864516 28954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:57.844263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:45601 (local address 127.12.34.195:53649)
0926 15:12:57.844977 (+ 714us) negotiation.cc:107] Waiting for socket to connect
0926 15:12:57.845020 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 15:12:57.847410 (+ 2390us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:12:57.848491 (+ 1081us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:12:57.848499 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:12:57.849016 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:12:57.849646 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:12:57.849663 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:12:57.850852 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:12:57.850859 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:12:57.852161 (+ 1302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:12:57.852173 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:12:57.853119 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:12:57.853131 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:12:57.853557 (+ 426us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:12:57.854254 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:12:57.854282 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:12:57.856631 (+ 2349us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:12:57.860122 (+ 3491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:12:57.860131 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:12:57.860146 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:12:57.860570 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:12:57.860967 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:12:57.860974 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:12:57.860978 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:12:57.861257 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:12:57.861931 (+ 674us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:12:57.861940 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:12:57.862361 (+ 421us) client_negotiation.cc:769] Sending connection context
0926 15:12:57.862715 (+ 354us) client_negotiation.cc:240] Negotiation successful
0926 15:12:57.863246 (+ 531us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":281,"threads_started":1}
I20260926 15:12:57.866039 28952 heartbeater.cc:347] Connected to a master server at 127.12.34.254:45601
I20260926 15:12:57.866341 28952 heartbeater.cc:464] Registering TS with master...
I20260926 15:12:57.867058 28952 heartbeater.cc:511] Master 127.12.34.254:45601 requested a full tablet report, sending...
I20260926 15:12:57.868394 28503 ts_manager.cc:194] Registered new tserver with Master: 6e3b1a3f8e1343239a873c89195df725 (127.12.34.195:44121)
I20260926 15:12:57.869261 28503 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.34.195@KRBTEST.COM'} at 127.12.34.195:53649
I20260926 15:12:57.875993 12427 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 15:12:57.882355 28691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:57.876908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43898 (local address 127.12.34.254:45601)
0926 15:12:57.877145 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0926 15:12:57.877154 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 15:12:57.877495 (+ 341us) server_negotiation.cc:407] Connection header received
0926 15:12:57.877639 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:12:57.877643 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:12:57.877810 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:12:57.877936 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 15:12:57.878313 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:12:57.879125 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:12:57.880523 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:12:57.881012 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:12:57.881214 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:12:57.881650 (+ 436us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 15:12:57.881852 (+ 202us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0926 15:12:57.882127 (+ 275us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:43898: token expired
Metrics: {"server-negotiator.queue_time_us":64}
W20260926 15:12:57.882483 28691 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:43898: token expired
W20260926 15:12:57.883059 28424 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.12.34.254:45601)
I20260926 15:12:57.896517 28691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:57.883456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43906 (local address 127.12.34.254:45601)
0926 15:12:57.883607 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0926 15:12:57.883611 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:12:57.883624 (+ 13us) server_negotiation.cc:407] Connection header received
0926 15:12:57.883808 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:12:57.883812 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:12:57.883861 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:12:57.883935 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:12:57.885686 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:12:57.886388 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:12:57.887768 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:12:57.888153 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:12:57.888242 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:12:57.891378 (+ 3136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:12:57.891383 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:12:57.891386 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:12:57.891412 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:12:57.894089 (+ 2677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:12:57.894872 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:12:57.894876 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:12:57.894879 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:12:57.894995 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:12:57.895342 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:12:57.895345 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:12:57.895347 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:12:57.895663 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:12:57.895838 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0926 15:12:57.896119 (+ 281us) server_negotiation.cc:299] Negotiation successful
0926 15:12:57.896268 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 15:12:57.899674 28503 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43906:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 15:12:57.908279 28503 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 15:12:57.937949 28966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:57.920340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.195:44121 (local address 127.0.0.1:38150)
0926 15:12:57.929589 (+ 9249us) negotiation.cc:107] Waiting for socket to connect
0926 15:12:57.929612 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 15:12:57.930385 (+ 773us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:12:57.931043 (+ 658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:12:57.931048 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:12:57.931073 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:12:57.931456 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:12:57.931469 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:12:57.933453 (+ 1984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:12:57.933464 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:12:57.935089 (+ 1625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:12:57.935109 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:12:57.936806 (+ 1697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:12:57.936813 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:12:57.937017 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:12:57.937070 (+ 53us) client_negotiation.cc:769] Sending connection context
0926 15:12:57.937236 (+ 166us) client_negotiation.cc:240] Negotiation successful
0926 15:12:57.937430 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9090,"mutex_wait_us":5,"thread_start_us":9260,"threads_started":1}
I20260926 15:12:57.938784 28965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:57.921276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38150 (local address 127.12.34.195:44121)
0926 15:12:57.930196 (+ 8920us) server_negotiation.cc:206] Beginning negotiation
0926 15:12:57.930206 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 15:12:57.930240 (+ 34us) server_negotiation.cc:407] Connection header received
0926 15:12:57.930605 (+ 365us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:12:57.930615 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:12:57.930830 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:12:57.931057 (+ 227us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:12:57.931697 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:12:57.933288 (+ 1591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:12:57.935262 (+ 1974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:12:57.936577 (+ 1315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:12:57.938029 (+ 1452us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:12:57.938253 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0926 15:12:57.938322 (+ 69us) server_negotiation.cc:299] Negotiation successful
0926 15:12:57.938477 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8737,"thread_start_us":8581,"threads_started":1}
I20260926 15:12:57.939056 28964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:57.920746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33198 (local address 127.12.34.193:45901)
0926 15:12:57.929042 (+ 8296us) server_negotiation.cc:206] Beginning negotiation
0926 15:12:57.929054 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 15:12:57.929725 (+ 671us) server_negotiation.cc:407] Connection header received
0926 15:12:57.929851 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:12:57.929861 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:12:57.930118 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:12:57.930326 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:12:57.931293 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:12:57.932884 (+ 1591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:12:57.935723 (+ 2839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:12:57.937202 (+ 1479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:12:57.937323 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:12:57.937540 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0926 15:12:57.938582 (+ 1042us) server_negotiation.cc:299] Negotiation successful
0926 15:12:57.938744 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6900,"thread_start_us":6747,"threads_started":1}
I20260926 15:12:57.939308 28962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:57.920155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.193:45901 (local address 127.0.0.1:33198)
0926 15:12:57.920883 (+ 728us) negotiation.cc:107] Waiting for socket to connect
0926 15:12:57.920928 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 15:12:57.921130 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:12:57.930373 (+ 9243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:12:57.930381 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:12:57.930412 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:12:57.931060 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:12:57.931075 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:12:57.933021 (+ 1946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:12:57.933029 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:12:57.935561 (+ 2532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:12:57.935575 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:12:57.937705 (+ 2130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:12:57.937711 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:12:57.938026 (+ 315us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:12:57.938098 (+ 72us) client_negotiation.cc:769] Sending connection context
0926 15:12:57.938195 (+ 97us) client_negotiation.cc:240] Negotiation successful
0926 15:12:57.938323 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"mutex_wait_us":201,"thread_start_us":357,"threads_started":1}
I20260926 15:12:57.939807 28967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:57.922059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41518 (local address 127.12.34.194:38857)
0926 15:12:57.922737 (+ 678us) server_negotiation.cc:206] Beginning negotiation
0926 15:12:57.922748 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 15:12:57.930831 (+ 8083us) server_negotiation.cc:407] Connection header received
0926 15:12:57.931035 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:12:57.931041 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:12:57.931248 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:12:57.931434 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:12:57.932020 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:12:57.933496 (+ 1476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:12:57.935978 (+ 2482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:12:57.937129 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:12:57.937258 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:12:57.937463 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0926 15:12:57.938275 (+ 812us) server_negotiation.cc:299] Negotiation successful
0926 15:12:57.938821 (+ 546us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":477,"thread_start_us":304,"threads_started":1}
I20260926 15:12:57.940347 28963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:57.921765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.194:38857 (local address 127.0.0.1:41518)
0926 15:12:57.930704 (+ 8939us) negotiation.cc:107] Waiting for socket to connect
0926 15:12:57.930723 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 15:12:57.930869 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:12:57.931447 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:12:57.931454 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:12:57.931482 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:12:57.931856 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:12:57.931867 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:12:57.933636 (+ 1769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:12:57.933643 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:12:57.935834 (+ 2191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:12:57.935849 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:12:57.937768 (+ 1919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:12:57.937775 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:12:57.938094 (+ 319us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:12:57.938119 (+ 25us) client_negotiation.cc:769] Sending connection context
0926 15:12:57.940018 (+ 1899us) client_negotiation.cc:240] Negotiation successful
0926 15:12:57.940110 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8628,"mutex_wait_us":60,"thread_start_us":7531,"threads_started":1}
I20260926 15:12:57.943070 28887 tablet_service.cc:1467] Processing CreateTablet for tablet 2ac8cd68b7b54a72904c0fc01889a24d (DEFAULT_TABLE table=test-table [id=60634d53cfe94e55aa84385b1cdb6968]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:12:57.943070 28624 tablet_service.cc:1467] Processing CreateTablet for tablet 2ac8cd68b7b54a72904c0fc01889a24d (DEFAULT_TABLE table=test-table [id=60634d53cfe94e55aa84385b1cdb6968]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:12:57.943202 28756 tablet_service.cc:1467] Processing CreateTablet for tablet 2ac8cd68b7b54a72904c0fc01889a24d (DEFAULT_TABLE table=test-table [id=60634d53cfe94e55aa84385b1cdb6968]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:12:57.944423 28756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ac8cd68b7b54a72904c0fc01889a24d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:12:57.944460 28887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ac8cd68b7b54a72904c0fc01889a24d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:12:57.944473 28624 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ac8cd68b7b54a72904c0fc01889a24d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:12:57.955780 28968 tablet_bootstrap.cc:492] T 2ac8cd68b7b54a72904c0fc01889a24d P 9fda66f1b152460e8e0cb199f028a5c0: Bootstrap starting.
I20260926 15:12:57.959144 28968 tablet_bootstrap.cc:654] T 2ac8cd68b7b54a72904c0fc01889a24d P 9fda66f1b152460e8e0cb199f028a5c0: Neither blocks nor log segments found. Creating new log.
I20260926 15:12:57.960129 28968 log.cc:824] T 2ac8cd68b7b54a72904c0fc01889a24d P 9fda66f1b152460e8e0cb199f028a5c0: Log is configured to *not* fsync() on all Append() calls
I20260926 15:12:57.962560 28689 heartbeater.cc:503] Master 127.12.34.254:45601 was elected leader, sending a full tablet report...
I20260926 15:12:57.962993 28968 tablet_bootstrap.cc:492] T 2ac8cd68b7b54a72904c0fc01889a24d P 9fda66f1b152460e8e0cb199f028a5c0: No bootstrap required, opened a new log
I20260926 15:12:57.963266 28968 ts_tablet_manager.cc:1397] T 2ac8cd68b7b54a72904c0fc01889a24d P 9fda66f1b152460e8e0cb199f028a5c0: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260926 15:12:57.965771 28969 tablet_bootstrap.cc:492] T 2ac8cd68b7b54a72904c0fc01889a24d P a311dd73aa814f63b863f901521cdcea: Bootstrap starting.
I20260926 15:12:57.968510 28969 tablet_bootstrap.cc:654] T 2ac8cd68b7b54a72904c0fc01889a24d P a311dd73aa814f63b863f901521cdcea: Neither blocks nor log segments found. Creating new log.
I20260926 15:12:57.968808 28970 tablet_bootstrap.cc:492] T 2ac8cd68b7b54a72904c0fc01889a24d P 6e3b1a3f8e1343239a873c89195df725: Bootstrap starting.
I20260926 15:12:57.969755 28969 log.cc:824] T 2ac8cd68b7b54a72904c0fc01889a24d P a311dd73aa814f63b863f901521cdcea: Log is configured to *not* fsync() on all Append() calls
I20260926 15:12:57.972262 28970 tablet_bootstrap.cc:654] T 2ac8cd68b7b54a72904c0fc01889a24d P 6e3b1a3f8e1343239a873c89195df725: Neither blocks nor log segments found. Creating new log.
I20260926 15:12:57.972107 28968 raft_consensus.cc:348] T 2ac8cd68b7b54a72904c0fc01889a24d P 9fda66f1b152460e8e0cb199f028a5c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a311dd73aa814f63b863f901521cdcea" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 45901 } } peers { permanent_uuid: "6e3b1a3f8e1343239a873c89195df725" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 44121 } } peers { permanent_uuid: "9fda66f1b152460e8e0cb199f028a5c0" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 38857 } }
I20260926 15:12:57.972479 28969 tablet_bootstrap.cc:492] T 2ac8cd68b7b54a72904c0fc01889a24d P a311dd73aa814f63b863f901521cdcea: No bootstrap required, opened a new log
I20260926 15:12:57.972645 28968 raft_consensus.cc:374] T 2ac8cd68b7b54a72904c0fc01889a24d P 9fda66f1b152460e8e0cb199f028a5c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:12:57.972748 28969 ts_tablet_manager.cc:1397] T 2ac8cd68b7b54a72904c0fc01889a24d P a311dd73aa814f63b863f901521cdcea: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260926 15:12:57.972760 28968 raft_consensus.cc:729] T 2ac8cd68b7b54a72904c0fc01889a24d P 9fda66f1b152460e8e0cb199f028a5c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fda66f1b152460e8e0cb199f028a5c0, State: Initialized, Role: FOLLOWER
I20260926 15:12:57.973484 28970 log.cc:824] T 2ac8cd68b7b54a72904c0fc01889a24d P 6e3b1a3f8e1343239a873c89195df725: Log is configured to *not* fsync() on all Append() calls
I20260926 15:12:57.973434 28968 consensus_queue.cc:260] T 2ac8cd68b7b54a72904c0fc01889a24d P 9fda66f1b152460e8e0cb199f028a5c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a311dd73aa814f63b863f901521cdcea" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 45901 } } peers { permanent_uuid: "6e3b1a3f8e1343239a873c89195df725" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 44121 } } peers { permanent_uuid: "9fda66f1b152460e8e0cb199f028a5c0" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 38857 } }
I20260926 15:12:57.974530 28821 heartbeater.cc:503] Master 127.12.34.254:45601 was elected leader, sending a full tablet report...
I20260926 15:12:57.975421 28968 ts_tablet_manager.cc:1428] T 2ac8cd68b7b54a72904c0fc01889a24d P 9fda66f1b152460e8e0cb199f028a5c0: Time spent starting tablet: real 0.012s user 0.013s sys 0.000s
I20260926 15:12:57.977906 28970 tablet_bootstrap.cc:492] T 2ac8cd68b7b54a72904c0fc01889a24d P 6e3b1a3f8e1343239a873c89195df725: No bootstrap required, opened a new log
I20260926 15:12:57.978166 28970 ts_tablet_manager.cc:1397] T 2ac8cd68b7b54a72904c0fc01889a24d P 6e3b1a3f8e1343239a873c89195df725: Time spent bootstrapping tablet: real 0.017s user 0.006s sys 0.003s
I20260926 15:12:57.982031 28969 raft_consensus.cc:348] T 2ac8cd68b7b54a72904c0fc01889a24d P a311dd73aa814f63b863f901521cdcea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a311dd73aa814f63b863f901521cdcea" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 45901 } } peers { permanent_uuid: "6e3b1a3f8e1343239a873c89195df725" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 44121 } } peers { permanent_uuid: "9fda66f1b152460e8e0cb199f028a5c0" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 38857 } }
I20260926 15:12:57.982403 28969 raft_consensus.cc:374] T 2ac8cd68b7b54a72904c0fc01889a24d P a311dd73aa814f63b863f901521cdcea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:12:57.982522 28969 raft_consensus.cc:729] T 2ac8cd68b7b54a72904c0fc01889a24d P a311dd73aa814f63b863f901521cdcea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a311dd73aa814f63b863f901521cdcea, State: Initialized, Role: FOLLOWER
I20260926 15:12:57.983080 28969 consensus_queue.cc:260] T 2ac8cd68b7b54a72904c0fc01889a24d P a311dd73aa814f63b863f901521cdcea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a311dd73aa814f63b863f901521cdcea" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 45901 } } peers { permanent_uuid: "6e3b1a3f8e1343239a873c89195df725" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 44121 } } peers { permanent_uuid: "9fda66f1b152460e8e0cb199f028a5c0" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 38857 } }
I20260926 15:12:57.984650 28969 ts_tablet_manager.cc:1428] T 2ac8cd68b7b54a72904c0fc01889a24d P a311dd73aa814f63b863f901521cdcea: Time spent starting tablet: real 0.012s user 0.013s sys 0.000s
I20260926 15:12:57.986537 28970 raft_consensus.cc:348] T 2ac8cd68b7b54a72904c0fc01889a24d P 6e3b1a3f8e1343239a873c89195df725 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a311dd73aa814f63b863f901521cdcea" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 45901 } } peers { permanent_uuid: "6e3b1a3f8e1343239a873c89195df725" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 44121 } } peers { permanent_uuid: "9fda66f1b152460e8e0cb199f028a5c0" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 38857 } }
I20260926 15:12:57.986912 28970 raft_consensus.cc:374] T 2ac8cd68b7b54a72904c0fc01889a24d P 6e3b1a3f8e1343239a873c89195df725 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:12:57.987048 28970 raft_consensus.cc:729] T 2ac8cd68b7b54a72904c0fc01889a24d P 6e3b1a3f8e1343239a873c89195df725 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e3b1a3f8e1343239a873c89195df725, State: Initialized, Role: FOLLOWER
I20260926 15:12:57.987632 28970 consensus_queue.cc:260] T 2ac8cd68b7b54a72904c0fc01889a24d P 6e3b1a3f8e1343239a873c89195df725 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a311dd73aa814f63b863f901521cdcea" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 45901 } } peers { permanent_uuid: "6e3b1a3f8e1343239a873c89195df725" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 44121 } } peers { permanent_uuid: "9fda66f1b152460e8e0cb199f028a5c0" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 38857 } }
I20260926 15:12:57.988734 28952 heartbeater.cc:503] Master 127.12.34.254:45601 was elected leader, sending a full tablet report...
I20260926 15:12:57.989264 28970 ts_tablet_manager.cc:1428] T 2ac8cd68b7b54a72904c0fc01889a24d P 6e3b1a3f8e1343239a873c89195df725: Time spent starting tablet: real 0.011s user 0.011s sys 0.001s
W20260926 15:12:58.093138 28953 tablet.cc:2369] T 2ac8cd68b7b54a72904c0fc01889a24d P 6e3b1a3f8e1343239a873c89195df725: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 15:12:58.110046 28973 raft_consensus.cc:482] T 2ac8cd68b7b54a72904c0fc01889a24d P 9fda66f1b152460e8e0cb199f028a5c0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 15:12:58.110317 28973 raft_consensus.cc:504] T 2ac8cd68b7b54a72904c0fc01889a24d P 9fda66f1b152460e8e0cb199f028a5c0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a311dd73aa814f63b863f901521cdcea" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 45901 } } peers { permanent_uuid: "6e3b1a3f8e1343239a873c89195df725" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 44121 } } peers { permanent_uuid: "9fda66f1b152460e8e0cb199f028a5c0" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 38857 } }
I20260926 15:12:58.111977 28973 leader_election.cc:288] T 2ac8cd68b7b54a72904c0fc01889a24d P 9fda66f1b152460e8e0cb199f028a5c0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a311dd73aa814f63b863f901521cdcea (127.12.34.193:45901), 6e3b1a3f8e1343239a873c89195df725 (127.12.34.195:44121)
I20260926 15:12:58.118645 28964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:58.112410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.194:53411 (local address 127.12.34.193:45901)
0926 15:12:58.112649 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0926 15:12:58.112657 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 15:12:58.113474 (+ 817us) server_negotiation.cc:407] Connection header received
0926 15:12:58.113708 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:12:58.113712 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:12:58.113770 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:12:58.113874 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:12:58.114412 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:12:58.115134 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:12:58.116917 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:12:58.117739 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:12:58.117919 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:12:58.117999 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 15:12:58.118276 (+ 277us) server_negotiation.cc:299] Negotiation successful
0926 15:12:58.118395 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 15:12:58.118729 28965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:58.112621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.194:48263 (local address 127.12.34.195:44121)
0926 15:12:58.112865 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0926 15:12:58.112874 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 15:12:58.113552 (+ 678us) server_negotiation.cc:407] Connection header received
0926 15:12:58.113742 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:12:58.113746 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:12:58.113813 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:12:58.113912 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:12:58.114567 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:12:58.115251 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:12:58.116988 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:12:58.117822 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:12:58.118010 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:12:58.118082 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 15:12:58.118359 (+ 277us) server_negotiation.cc:299] Negotiation successful
0926 15:12:58.118480 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260926 15:12:58.118824 28977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:58.112350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.193:45901 (local address 127.12.34.194:53411)
0926 15:12:58.113356 (+ 1006us) negotiation.cc:107] Waiting for socket to connect
0926 15:12:58.113377 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 15:12:58.113549 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:12:58.113886 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:12:58.113891 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:12:58.113914 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:12:58.114246 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:12:58.114255 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:12:58.115267 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:12:58.115274 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:12:58.116757 (+ 1483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:12:58.116769 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:12:58.117965 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:12:58.117971 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:12:58.118119 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:12:58.118142 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 15:12:58.118250 (+ 108us) client_negotiation.cc:240] Negotiation successful
0926 15:12:58.118381 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":847,"mutex_wait_us":22,"thread_start_us":513,"threads_started":1}
I20260926 15:12:58.118880 28978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:58.112477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.195:44121 (local address 127.12.34.194:48263)
0926 15:12:58.113467 (+ 990us) negotiation.cc:107] Waiting for socket to connect
0926 15:12:58.113481 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 15:12:58.113584 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:12:58.113924 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:12:58.113928 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:12:58.113951 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:12:58.114411 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:12:58.114420 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:12:58.115383 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:12:58.115391 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:12:58.116829 (+ 1438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:12:58.116841 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:12:58.118042 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:12:58.118047 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:12:58.118199 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:12:58.118221 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 15:12:58.118334 (+ 113us) client_negotiation.cc:240] Negotiation successful
0926 15:12:58.118452 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":881,"mutex_wait_us":1,"thread_start_us":682,"threads_started":1}
I20260926 15:12:58.119835 28907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ac8cd68b7b54a72904c0fc01889a24d" candidate_uuid: "9fda66f1b152460e8e0cb199f028a5c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e3b1a3f8e1343239a873c89195df725" is_pre_election: true
I20260926 15:12:58.119843 28644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ac8cd68b7b54a72904c0fc01889a24d" candidate_uuid: "9fda66f1b152460e8e0cb199f028a5c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a311dd73aa814f63b863f901521cdcea" is_pre_election: true
I20260926 15:12:58.120299 28644 raft_consensus.cc:2454] T 2ac8cd68b7b54a72904c0fc01889a24d P a311dd73aa814f63b863f901521cdcea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9fda66f1b152460e8e0cb199f028a5c0 in term 0.
I20260926 15:12:58.120316 28907 raft_consensus.cc:2454] T 2ac8cd68b7b54a72904c0fc01889a24d P 6e3b1a3f8e1343239a873c89195df725 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9fda66f1b152460e8e0cb199f028a5c0 in term 0.
I20260926 15:12:58.120992 28712 leader_election.cc:302] T 2ac8cd68b7b54a72904c0fc01889a24d P 9fda66f1b152460e8e0cb199f028a5c0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 6e3b1a3f8e1343239a873c89195df725, 9fda66f1b152460e8e0cb199f028a5c0, a311dd73aa814f63b863f901521cdcea; no voters:
I20260926 15:12:58.121462 28973 raft_consensus.cc:2781] T 2ac8cd68b7b54a72904c0fc01889a24d P 9fda66f1b152460e8e0cb199f028a5c0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 15:12:58.121601 28973 raft_consensus.cc:482] T 2ac8cd68b7b54a72904c0fc01889a24d P 9fda66f1b152460e8e0cb199f028a5c0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 15:12:58.121707 28973 raft_consensus.cc:3037] T 2ac8cd68b7b54a72904c0fc01889a24d P 9fda66f1b152460e8e0cb199f028a5c0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:12:58.124084 28973 raft_consensus.cc:504] T 2ac8cd68b7b54a72904c0fc01889a24d P 9fda66f1b152460e8e0cb199f028a5c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a311dd73aa814f63b863f901521cdcea" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 45901 } } peers { permanent_uuid: "6e3b1a3f8e1343239a873c89195df725" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 44121 } } peers { permanent_uuid: "9fda66f1b152460e8e0cb199f028a5c0" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 38857 } }
I20260926 15:12:58.124709 28973 leader_election.cc:288] T 2ac8cd68b7b54a72904c0fc01889a24d P 9fda66f1b152460e8e0cb199f028a5c0 [CANDIDATE]: Term 1 election: Requested vote from peers a311dd73aa814f63b863f901521cdcea (127.12.34.193:45901), 6e3b1a3f8e1343239a873c89195df725 (127.12.34.195:44121)
I20260926 15:12:58.125165 28644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ac8cd68b7b54a72904c0fc01889a24d" candidate_uuid: "9fda66f1b152460e8e0cb199f028a5c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a311dd73aa814f63b863f901521cdcea"
I20260926 15:12:58.125308 28907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ac8cd68b7b54a72904c0fc01889a24d" candidate_uuid: "9fda66f1b152460e8e0cb199f028a5c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e3b1a3f8e1343239a873c89195df725"
I20260926 15:12:58.125385 28644 raft_consensus.cc:3037] T 2ac8cd68b7b54a72904c0fc01889a24d P a311dd73aa814f63b863f901521cdcea [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:12:58.125519 28907 raft_consensus.cc:3037] T 2ac8cd68b7b54a72904c0fc01889a24d P 6e3b1a3f8e1343239a873c89195df725 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:12:58.128684 28644 raft_consensus.cc:2454] T 2ac8cd68b7b54a72904c0fc01889a24d P a311dd73aa814f63b863f901521cdcea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9fda66f1b152460e8e0cb199f028a5c0 in term 1.
I20260926 15:12:58.128684 28907 raft_consensus.cc:2454] T 2ac8cd68b7b54a72904c0fc01889a24d P 6e3b1a3f8e1343239a873c89195df725 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9fda66f1b152460e8e0cb199f028a5c0 in term 1.
I20260926 15:12:58.129145 28711 leader_election.cc:302] T 2ac8cd68b7b54a72904c0fc01889a24d P 9fda66f1b152460e8e0cb199f028a5c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9fda66f1b152460e8e0cb199f028a5c0, a311dd73aa814f63b863f901521cdcea; no voters:
I20260926 15:12:58.129516 28973 raft_consensus.cc:2781] T 2ac8cd68b7b54a72904c0fc01889a24d P 9fda66f1b152460e8e0cb199f028a5c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:12:58.129998 28973 raft_consensus.cc:686] T 2ac8cd68b7b54a72904c0fc01889a24d P 9fda66f1b152460e8e0cb199f028a5c0 [term 1 LEADER]: Becoming Leader. State: Replica: 9fda66f1b152460e8e0cb199f028a5c0, State: Running, Role: LEADER
I20260926 15:12:58.130519 28973 consensus_queue.cc:237] T 2ac8cd68b7b54a72904c0fc01889a24d P 9fda66f1b152460e8e0cb199f028a5c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a311dd73aa814f63b863f901521cdcea" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 45901 } } peers { permanent_uuid: "6e3b1a3f8e1343239a873c89195df725" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 44121 } } peers { permanent_uuid: "9fda66f1b152460e8e0cb199f028a5c0" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 38857 } }
I20260926 15:12:58.135006 28503 catalog_manager.cc:5495] T 2ac8cd68b7b54a72904c0fc01889a24d P 9fda66f1b152460e8e0cb199f028a5c0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9fda66f1b152460e8e0cb199f028a5c0 (127.12.34.194). New cstate: current_term: 1 leader_uuid: "9fda66f1b152460e8e0cb199f028a5c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a311dd73aa814f63b863f901521cdcea" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 45901 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e3b1a3f8e1343239a873c89195df725" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 44121 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9fda66f1b152460e8e0cb199f028a5c0" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 38857 } health_report { overall_health: HEALTHY } } }
W20260926 15:12:58.144635 28822 tablet.cc:2369] T 2ac8cd68b7b54a72904c0fc01889a24d P 9fda66f1b152460e8e0cb199f028a5c0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 15:12:58.167042 28967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:58.161939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41534 (local address 127.12.34.194:38857)
0926 15:12:58.162142 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0926 15:12:58.162148 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:12:58.162167 (+ 19us) server_negotiation.cc:407] Connection header received
0926 15:12:58.162328 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:12:58.162332 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:12:58.162407 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:12:58.162553 (+ 146us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 15:12:58.162926 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:12:58.163609 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:12:58.164940 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:12:58.165402 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:12:58.165533 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:12:58.165994 (+ 461us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 15:12:58.166464 (+ 470us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 15:12:58.166625 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0926 15:12:58.166680 (+ 55us) server_negotiation.cc:299] Negotiation successful
0926 15:12:58.166776 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20260926 15:12:58.169270 28690 tablet.cc:2369] T 2ac8cd68b7b54a72904c0fc01889a24d P a311dd73aa814f63b863f901521cdcea: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 15:12:58.175454 28907 raft_consensus.cc:1261] T 2ac8cd68b7b54a72904c0fc01889a24d P 6e3b1a3f8e1343239a873c89195df725 [term 1 FOLLOWER]: Refusing update from remote peer 9fda66f1b152460e8e0cb199f028a5c0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:12:58.175496 28644 raft_consensus.cc:1261] T 2ac8cd68b7b54a72904c0fc01889a24d P a311dd73aa814f63b863f901521cdcea [term 1 FOLLOWER]: Refusing update from remote peer 9fda66f1b152460e8e0cb199f028a5c0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:12:58.176447 28979 consensus_queue.cc:1035] T 2ac8cd68b7b54a72904c0fc01889a24d P 9fda66f1b152460e8e0cb199f028a5c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a311dd73aa814f63b863f901521cdcea" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 45901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:12:58.176781 28973 consensus_queue.cc:1035] T 2ac8cd68b7b54a72904c0fc01889a24d P 9fda66f1b152460e8e0cb199f028a5c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e3b1a3f8e1343239a873c89195df725" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 44121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:12:58.188313 28982 mvcc.cc:205] Tried to move back new op lower bound from 7333624128197054464 to 7333624128026648576. Current Snapshot: MvccSnapshot[applied={T|T < 7333624128197054464}]
I20260926 15:12:58.203891 28502 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43906:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 15:12:58.204146 28502 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43906:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 15:12:58.208076 28502 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7756b74c7007403c85a266f55693c0bc: Sending DeleteTablet for 3 replicas of tablet 2ac8cd68b7b54a72904c0fc01889a24d
I20260926 15:12:58.209293 28756 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ac8cd68b7b54a72904c0fc01889a24d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:12:58 UTC) from {username='kudu', principal='kudu/127.12.34.254@KRBTEST.COM'} at 127.0.0.1:41518
I20260926 15:12:58.209290 28887 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ac8cd68b7b54a72904c0fc01889a24d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:12:58 UTC) from {username='kudu', principal='kudu/127.12.34.254@KRBTEST.COM'} at 127.0.0.1:38150
I20260926 15:12:58.209486 12427 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 28540
I20260926 15:12:58.210207 28624 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ac8cd68b7b54a72904c0fc01889a24d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:12:58 UTC) from {username='kudu', principal='kudu/127.12.34.254@KRBTEST.COM'} at 127.0.0.1:33198
I20260926 15:12:58.212034 28991 tablet_replica.cc:330] T 2ac8cd68b7b54a72904c0fc01889a24d P 9fda66f1b152460e8e0cb199f028a5c0: stopping tablet replica
I20260926 15:12:58.212628 28991 raft_consensus.cc:2229] T 2ac8cd68b7b54a72904c0fc01889a24d P 9fda66f1b152460e8e0cb199f028a5c0 [term 1 LEADER]: Raft consensus shutting down.
I20260926 15:12:58.215555 28991 raft_consensus.cc:2258] T 2ac8cd68b7b54a72904c0fc01889a24d P 9fda66f1b152460e8e0cb199f028a5c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:12:58.220057 28991 ts_tablet_manager.cc:1905] T 2ac8cd68b7b54a72904c0fc01889a24d P 9fda66f1b152460e8e0cb199f028a5c0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:12:58.221560 28993 tablet_replica.cc:330] T 2ac8cd68b7b54a72904c0fc01889a24d P a311dd73aa814f63b863f901521cdcea: stopping tablet replica
I20260926 15:12:58.222232 28993 raft_consensus.cc:2229] T 2ac8cd68b7b54a72904c0fc01889a24d P a311dd73aa814f63b863f901521cdcea [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 15:12:58.223315 28993 raft_consensus.cc:2258] T 2ac8cd68b7b54a72904c0fc01889a24d P a311dd73aa814f63b863f901521cdcea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:12:58.226642 28993 ts_tablet_manager.cc:1905] T 2ac8cd68b7b54a72904c0fc01889a24d P a311dd73aa814f63b863f901521cdcea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:12:58.227164 28992 tablet_replica.cc:330] T 2ac8cd68b7b54a72904c0fc01889a24d P 6e3b1a3f8e1343239a873c89195df725: stopping tablet replica
I20260926 15:12:58.227645 28992 raft_consensus.cc:2229] T 2ac8cd68b7b54a72904c0fc01889a24d P 6e3b1a3f8e1343239a873c89195df725 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 15:12:58.228225 28992 raft_consensus.cc:2258] T 2ac8cd68b7b54a72904c0fc01889a24d P 6e3b1a3f8e1343239a873c89195df725 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:12:58.230443 28992 ts_tablet_manager.cc:1905] T 2ac8cd68b7b54a72904c0fc01889a24d P 6e3b1a3f8e1343239a873c89195df725: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:12:58.231825 28964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:58.210751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33202 (local address 127.12.34.193:45901)
0926 15:12:58.210951 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0926 15:12:58.210957 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:12:58.210974 (+ 17us) server_negotiation.cc:407] Connection header received
0926 15:12:58.211030 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:12:58.211035 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:12:58.211110 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:12:58.211226 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:12:58.213429 (+ 2203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:12:58.215171 (+ 1742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:12:58.217786 (+ 2615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:12:58.218862 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:12:58.219038 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:12:58.223459 (+ 4421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:12:58.223476 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:12:58.223494 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:12:58.223535 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:12:58.227539 (+ 4004us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:12:58.228427 (+ 888us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:12:58.228436 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:12:58.228440 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:12:58.228619 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:12:58.229118 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:12:58.229123 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:12:58.229126 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:12:58.229609 (+ 483us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:12:58.229919 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0926 15:12:58.231168 (+ 1249us) server_negotiation.cc:299] Negotiation successful
0926 15:12:58.231407 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 15:12:58.232671 28684 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:12:58.234534 28991 ts_tablet_manager.cc:1918] T 2ac8cd68b7b54a72904c0fc01889a24d P 9fda66f1b152460e8e0cb199f028a5c0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 15:12:58.234901 28991 log.cc:1196] T 2ac8cd68b7b54a72904c0fc01889a24d P 9fda66f1b152460e8e0cb199f028a5c0: Deleting WAL directory at /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/ts-1/wal/wals/2ac8cd68b7b54a72904c0fc01889a24d
I20260926 15:12:58.235769 28991 ts_tablet_manager.cc:1939] T 2ac8cd68b7b54a72904c0fc01889a24d P 9fda66f1b152460e8e0cb199f028a5c0: Deleting consensus metadata
I20260926 15:12:58.238070 28489 catalog_manager.cc:4841] TS 9fda66f1b152460e8e0cb199f028a5c0 (127.12.34.194:38857): tablet 2ac8cd68b7b54a72904c0fc01889a24d (table test-table [id=60634d53cfe94e55aa84385b1cdb6968]) successfully deleted
I20260926 15:12:58.238636 28992 ts_tablet_manager.cc:1918] T 2ac8cd68b7b54a72904c0fc01889a24d P 6e3b1a3f8e1343239a873c89195df725: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 15:12:58.238852 28992 log.cc:1196] T 2ac8cd68b7b54a72904c0fc01889a24d P 6e3b1a3f8e1343239a873c89195df725: Deleting WAL directory at /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/ts-2/wal/wals/2ac8cd68b7b54a72904c0fc01889a24d
I20260926 15:12:58.239428 28992 ts_tablet_manager.cc:1939] T 2ac8cd68b7b54a72904c0fc01889a24d P 6e3b1a3f8e1343239a873c89195df725: Deleting consensus metadata
I20260926 15:12:58.241230 28490 catalog_manager.cc:4841] TS 6e3b1a3f8e1343239a873c89195df725 (127.12.34.195:44121): tablet 2ac8cd68b7b54a72904c0fc01889a24d (table test-table [id=60634d53cfe94e55aa84385b1cdb6968]) successfully deleted
I20260926 15:12:58.325469 12427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8uoJGv/build/asan/bin/kudu with pid 28540
I20260926 15:12:58.342931 12427 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 28693
W20260926 15:12:58.342906 28489 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.34.193:45901 (error 108)
I20260926 15:12:58.344825 28963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:58.344361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.193:45901 (local address 127.0.0.1:33206)
0926 15:12:58.344541 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0926 15:12:58.344644 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.193:45901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260926 15:12:58.346139 28489 catalog_manager.cc:4584] TS a311dd73aa814f63b863f901521cdcea (127.12.34.193:45901): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2ac8cd68b7b54a72904c0fc01889a24d: Network error: Client connection negotiation failed: client connection to 127.12.34.193:45901: connect: Connection refused (error 111)
I20260926 15:12:58.358946 28967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:58.343986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41544 (local address 127.12.34.194:38857)
0926 15:12:58.344171 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0926 15:12:58.344177 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:12:58.344196 (+ 19us) server_negotiation.cc:407] Connection header received
0926 15:12:58.344263 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:12:58.344269 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:12:58.344344 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:12:58.344449 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:12:58.346071 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:12:58.347338 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:12:58.348602 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:12:58.349326 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:12:58.349454 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:12:58.351863 (+ 2409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:12:58.351881 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:12:58.351901 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:12:58.351945 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:12:58.355916 (+ 3971us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:12:58.356604 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:12:58.356614 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:12:58.356618 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:12:58.356783 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:12:58.357391 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:12:58.357397 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:12:58.357401 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:12:58.357824 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:12:58.358150 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0926 15:12:58.358342 (+ 192us) server_negotiation.cc:299] Negotiation successful
0926 15:12:58.358560 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 15:12:58.359769 28816 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:12:58.399825 28963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:58.399396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.193:45901 (local address 127.0.0.1:33210)
0926 15:12:58.399587 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0926 15:12:58.399685 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.193:45901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260926 15:12:58.400918 28963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:58.400584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.193:45901 (local address 127.0.0.1:33224)
0926 15:12:58.400708 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0926 15:12:58.400787 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.193:45901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260926 15:12:58.461735 28963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:58.461024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.193:45901 (local address 127.0.0.1:33236)
0926 15:12:58.461396 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0926 15:12:58.461531 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.193:45901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182}
I20260926 15:12:58.463169 28963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:58.462689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.193:45901 (local address 127.0.0.1:33244)
0926 15:12:58.462899 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0926 15:12:58.463005 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.193:45901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260926 15:12:58.483034 12427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8uoJGv/build/asan/bin/kudu with pid 28693
I20260926 15:12:58.501600 12427 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 28824
I20260926 15:12:58.515750 28965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:58.502449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38158 (local address 127.12.34.195:44121)
0926 15:12:58.502624 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0926 15:12:58.502629 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:12:58.502647 (+ 18us) server_negotiation.cc:407] Connection header received
0926 15:12:58.502832 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:12:58.502837 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:12:58.502913 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:12:58.503003 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:12:58.504320 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:12:58.505065 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:12:58.506463 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:12:58.506882 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:12:58.507009 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:12:58.509757 (+ 2748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:12:58.509769 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:12:58.509784 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:12:58.509815 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:12:58.512709 (+ 2894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:12:58.513600 (+ 891us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:12:58.513607 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:12:58.513609 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:12:58.513757 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:12:58.514173 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:12:58.514176 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:12:58.514178 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:12:58.514535 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:12:58.514768 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0926 15:12:58.515223 (+ 455us) server_negotiation.cc:299] Negotiation successful
0926 15:12:58.515383 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 15:12:58.516453 28947 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:12:58.543605 28963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:58.543164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.193:45901 (local address 127.0.0.1:33248)
0926 15:12:58.543367 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0926 15:12:58.543447 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.193:45901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260926 15:12:58.544569 28963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:58.544219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.193:45901 (local address 127.0.0.1:33252)
0926 15:12:58.544410 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0926 15:12:58.544474 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.193:45901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260926 15:12:58.611526 12427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8uoJGv/build/asan/bin/kudu with pid 28824
I20260926 15:12:58.628495 12427 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 28470
I20260926 15:12:58.629688 28533 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:12:58.746520 28963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:12:58.745818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.193:45901 (local address 127.0.0.1:33254)
0926 15:12:58.746146 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0926 15:12:58.746294 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.193:45901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168}
I20260926 15:12:58.746603 12427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8uoJGv/build/asan/bin/kudu with pid 28470
I20260926 15:13:18.764972 12427 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8uoJGv/build/asan/bin/kudu
/tmp/dist-test-task8uoJGv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.34.254:45601
--webserver_interface=127.12.34.254
--webserver_port=43841
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.254.keytab
--principal=kudu/127.12.34.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.34.212:40893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.34.254:45601
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:13:19.091598 28999 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:13:19.091943 28999 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:13:19.092123 28999 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:13:19.099747 28999 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 15:13:19.099879 28999 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 15:13:19.099949 28999 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 15:13:19.100028 28999 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:13:19.100109 28999 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:13:19.100207 28999 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:13:19.100297 28999 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:13:19.100376 28999 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:13:19.110077 28999 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.34.212:40893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.12.34.254:45601
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.254.keytab
--principal=kudu/127.12.34.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.34.254:45601
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.34.254
--webserver_port=43841
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:13:19.111380 28999 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:13:19.112632 28999 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:13:19.124779 29007 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:13:19.124804 29004 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:13:19.124908 29005 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:13:19.125758 28999 server_base.cc:1034] running on GCE node
I20260926 15:13:19.126539 28999 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:13:19.127992 28999 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:13:19.129205 28999 hybrid_clock.cc:648] HybridClock initialized: now 1790435599129153 us; error 36 us; skew 500 ppm
I20260926 15:13:19.129554 28999 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:13:19 dist-test-slave-865v krb5kdc[27919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435599, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:13:19.141993 28999 init.cc:377] Logged in from keytab as kudu/127.12.34.254@KRBTEST.COM (short username kudu)
I20260926 15:13:19.143532 28999 webserver.cc:460] Webserver started at http://127.12.34.254:43841/ using document root <none> and password file <none>
I20260926 15:13:19.144227 28999 fs_manager.cc:359] Metadata directory not provided
I20260926 15:13:19.144366 28999 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:13:19.149668 28999 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260926 15:13:19.152295 29014 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:13:19.153662 28999 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 15:13:19.153863 28999 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/master-0/data,/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/master-0/wal
uuid: "7756b74c7007403c85a266f55693c0bc"
format_stamp: "Formatted at 2026-09-26 15:12:33 on dist-test-slave-865v"
server_key: "3adac6befea70a01de26d053623217b3"
server_key_iv: "eb96e0558a0965c6ed14ad592c0b6369"
server_key_version: "encryptionkey@0"
I20260926 15:13:19.154779 28999 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:13:19.169768 28999 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:13:19.173629 28999 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:13:19.173913 28999 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:13:19.192271 28999 rpc_server.cc:307] RPC server started. Bound to: 127.12.34.254:45601
I20260926 15:13:19.192307 29065 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.34.254:45601 every 8 connection(s)
I20260926 15:13:19.193742 28999 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
I20260926 15:13:19.198076 12427 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8uoJGv/build/asan/bin/kudu as pid 28999
I20260926 15:13:19.199084 12427 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8uoJGv/build/asan/bin/kudu
/tmp/dist-test-task8uoJGv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.34.193:45901
--local_ip_for_outbound_sockets=127.12.34.193
--tserver_master_addrs=127.12.34.254:45601
--webserver_port=36813
--webserver_interface=127.12.34.193
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.193.keytab
--principal=kudu/127.12.34.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.34.212:40893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 15:13:19.201195 29066 sys_catalog.cc:263] Verifying existing consensus state
I20260926 15:13:19.206195 29066 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7756b74c7007403c85a266f55693c0bc: Bootstrap starting.
I20260926 15:13:19.226824 29066 log.cc:824] T 00000000000000000000000000000000 P 7756b74c7007403c85a266f55693c0bc: Log is configured to *not* fsync() on all Append() calls
I20260926 15:13:19.243270 29066 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7756b74c7007403c85a266f55693c0bc: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 15:13:19.243834 29066 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7756b74c7007403c85a266f55693c0bc: Bootstrap complete.
I20260926 15:13:19.252856 29066 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7756b74c7007403c85a266f55693c0bc [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7756b74c7007403c85a266f55693c0bc" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 45601 } }
I20260926 15:13:19.253790 29066 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7756b74c7007403c85a266f55693c0bc [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7756b74c7007403c85a266f55693c0bc, State: Initialized, Role: FOLLOWER
I20260926 15:13:19.254384 29066 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7756b74c7007403c85a266f55693c0bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7756b74c7007403c85a266f55693c0bc" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 45601 } }
I20260926 15:13:19.254607 29066 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7756b74c7007403c85a266f55693c0bc [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 15:13:19.254735 29066 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7756b74c7007403c85a266f55693c0bc [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 15:13:19.254923 29066 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7756b74c7007403c85a266f55693c0bc [term 2 FOLLOWER]: Advancing to term 3
I20260926 15:13:19.259114 29066 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7756b74c7007403c85a266f55693c0bc [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7756b74c7007403c85a266f55693c0bc" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 45601 } }
I20260926 15:13:19.259610 29066 leader_election.cc:302] T 00000000000000000000000000000000 P 7756b74c7007403c85a266f55693c0bc [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7756b74c7007403c85a266f55693c0bc; no voters:
I20260926 15:13:19.260294 29066 leader_election.cc:288] T 00000000000000000000000000000000 P 7756b74c7007403c85a266f55693c0bc [CANDIDATE]: Term 3 election: Requested vote from peers
I20260926 15:13:19.260444 29070 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7756b74c7007403c85a266f55693c0bc [term 3 FOLLOWER]: Leader election won for term 3
I20260926 15:13:19.262605 29066 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7756b74c7007403c85a266f55693c0bc [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:13:19.267495 29070 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7756b74c7007403c85a266f55693c0bc [term 3 LEADER]: Becoming Leader. State: Replica: 7756b74c7007403c85a266f55693c0bc, State: Running, Role: LEADER
I20260926 15:13:19.268237 29070 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7756b74c7007403c85a266f55693c0bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7756b74c7007403c85a266f55693c0bc" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 45601 } }
I20260926 15:13:19.270881 29072 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7756b74c7007403c85a266f55693c0bc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7756b74c7007403c85a266f55693c0bc. Latest consensus state: current_term: 3 leader_uuid: "7756b74c7007403c85a266f55693c0bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7756b74c7007403c85a266f55693c0bc" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 45601 } } }
I20260926 15:13:19.271296 29072 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7756b74c7007403c85a266f55693c0bc [sys.catalog]: This master's current role is: LEADER
I20260926 15:13:19.271049 29071 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7756b74c7007403c85a266f55693c0bc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "7756b74c7007403c85a266f55693c0bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7756b74c7007403c85a266f55693c0bc" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 45601 } } }
I20260926 15:13:19.271451 29071 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7756b74c7007403c85a266f55693c0bc [sys.catalog]: This master's current role is: LEADER
I20260926 15:13:19.274596 29082 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:13:19.282435 29082 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 15:13:19.284312 29082 catalog_manager.cc:1259] Loaded cluster ID: dd2c37d372564116a5191e490b674674
I20260926 15:13:19.284493 29082 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 15:13:19.289196 29087 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 15:13:19.289880 29082 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:13:19.291246 29082 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 7756b74c7007403c85a266f55693c0bc: Loaded TSK: 0
I20260926 15:13:19.291390 29082 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 7756b74c7007403c85a266f55693c0bc: Loaded TSK: 1
F20260926 15:13:19.292177 29082 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7ff0f61889ad google::LogMessage::Fail() at ??:0
@ 0x7ff0f618b88a google::LogMessage::SendToLog() at ??:0
@ 0x7ff0f61884de google::LogMessage::Flush() at ??:0
@ 0x7ff0f6189a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7ff0f9eec7d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7ff0f9edb73e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7ff10bbb71c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7ff10bbb6783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7ff10bc422a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7ff10bc420c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7ff10bc8c5e9 std::function<>::operator()() at ??:0
@ 0x7ff10bbbacdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7ff10bbb8c36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7ff10bc40610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7ff10bc40439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7ff0f8dd4345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7ff0f8dd791c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7ff0f8dd7799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7ff0f8db2eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7ff0fc08c6db start_thread at ??:0
@ 0x7ff0f502261f clone at ??:0
W20260926 15:13:19.569984 29068 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:13:19.570298 29068 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:13:19.570470 29068 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:13:19.577750 29068 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:13:19.577915 29068 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:13:19.578096 29068 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.34.193
I20260926 15:13:19.587925 29068 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.34.212:40893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.193.keytab
--principal=kudu/127.12.34.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.34.193:45901
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.34.193
--webserver_port=36813
--tserver_master_addrs=127.12.34.254:45601
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.34.193
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:13:19.589228 29068 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:13:19.590457 29068 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:13:19.603226 29113 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:13:19.603488 29116 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:13:19.603554 29114 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:13:19.604315 29068 server_base.cc:1034] running on GCE node
I20260926 15:13:19.604919 29068 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:13:19.605865 29068 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:13:19.607079 29068 hybrid_clock.cc:648] HybridClock initialized: now 1790435599607038 us; error 30 us; skew 500 ppm
I20260926 15:13:19.607421 29068 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:13:19 dist-test-slave-865v krb5kdc[27919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435599, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:13:19.610546 29068 init.cc:377] Logged in from keytab as kudu/127.12.34.193@KRBTEST.COM (short username kudu)
I20260926 15:13:19.611980 29068 webserver.cc:460] Webserver started at http://127.12.34.193:36813/ using document root <none> and password file <none>
I20260926 15:13:19.612643 29068 fs_manager.cc:359] Metadata directory not provided
I20260926 15:13:19.612793 29068 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:13:19.617967 29068 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 15:13:19.620887 29123 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:13:19.622216 29068 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 15:13:19.622413 29068 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/ts-0/data,/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/ts-0/wal
uuid: "a311dd73aa814f63b863f901521cdcea"
format_stamp: "Formatted at 2026-09-26 15:12:34 on dist-test-slave-865v"
server_key: "c274070bb383f9b2ce7d764574679349"
server_key_iv: "6e617dfd8ed626a53b3b93bba0428b34"
server_key_version: "encryptionkey@0"
I20260926 15:13:19.623340 29068 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:13:19.647819 29068 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:13:19.651877 29068 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:13:19.652151 29068 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:13:19.653120 29068 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:13:19.654844 29130 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
W20260926 15:13:19.659953 29130 ts_tablet_manager.cc:1852] T 2ac8cd68b7b54a72904c0fc01889a24d P a311dd73aa814f63b863f901521cdcea: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260926 15:13:19.660910 29130 ts_tablet_manager.cc:1905] T 2ac8cd68b7b54a72904c0fc01889a24d P a311dd73aa814f63b863f901521cdcea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:13:19.668040 29130 ts_tablet_manager.cc:1918] T 2ac8cd68b7b54a72904c0fc01889a24d P a311dd73aa814f63b863f901521cdcea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 15:13:19.668242 29130 log.cc:1196] T 2ac8cd68b7b54a72904c0fc01889a24d P a311dd73aa814f63b863f901521cdcea: Deleting WAL directory at /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/ts-0/wal/wals/2ac8cd68b7b54a72904c0fc01889a24d
I20260926 15:13:19.668728 29130 ts_tablet_manager.cc:1939] T 2ac8cd68b7b54a72904c0fc01889a24d P a311dd73aa814f63b863f901521cdcea: Deleting consensus metadata
I20260926 15:13:19.669184 29068 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260926 15:13:19.669318 29068 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.015s user 0.000s sys 0.001s
I20260926 15:13:19.669502 29068 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:13:19.669606 29068 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:13:19.703130 29068 rpc_server.cc:307] RPC server started. Bound to: 127.12.34.193:45901
I20260926 15:13:19.703187 29236 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.34.193:45901 every 8 connection(s)
I20260926 15:13:19.704423 29068 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/ts-0/data/info.pb
I20260926 15:13:19.705015 12427 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8uoJGv/build/asan/bin/kudu as pid 29068
I20260926 15:13:19.705883 12427 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8uoJGv/build/asan/bin/kudu
/tmp/dist-test-task8uoJGv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.34.194:38857
--local_ip_for_outbound_sockets=127.12.34.194
--tserver_master_addrs=127.12.34.254:45601
--webserver_port=36799
--webserver_interface=127.12.34.194
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.194.keytab
--principal=kudu/127.12.34.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.34.212:40893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 15:13:19.709388 29239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:19.707423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:45601 (local address 127.12.34.193:51619)
0926 15:13:19.708207 (+ 784us) negotiation.cc:107] Waiting for socket to connect
0926 15:13:19.708482 (+ 275us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.254:45601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":333,"threads_started":1}
W20260926 15:13:19.709959 29237 heartbeater.cc:644] Failed to heartbeat to 127.12.34.254:45601 (0 consecutive failures): Network error: Failed to ping master at 127.12.34.254:45601: Client connection negotiation failed: client connection to 127.12.34.254:45601: connect: Connection refused (error 111)
I20260926 15:13:19.710798 29239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:19.710506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:45601 (local address 127.12.34.193:32939)
0926 15:13:19.710639 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0926 15:13:19.710702 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.254:45601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260926 15:13:19.711673 29239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:19.711379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:45601 (local address 127.12.34.193:53375)
0926 15:13:19.711504 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0926 15:13:19.711569 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.254:45601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260926 15:13:19.711939 29237 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 15:13:20.026077 29240 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:13:20.026343 29240 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:13:20.026543 29240 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:13:20.033905 29240 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:13:20.034005 29240 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:13:20.034171 29240 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.34.194
I20260926 15:13:20.043624 29240 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.34.212:40893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.194.keytab
--principal=kudu/127.12.34.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.34.194:38857
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.34.194
--webserver_port=36799
--tserver_master_addrs=127.12.34.254:45601
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.34.194
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:13:20.044869 29240 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:13:20.046114 29240 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:13:20.058872 29246 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:13:20.058898 29245 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:13:20.058876 29248 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:13:20.059676 29240 server_base.cc:1034] running on GCE node
I20260926 15:13:20.060262 29240 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:13:20.061276 29240 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:13:20.062510 29240 hybrid_clock.cc:648] HybridClock initialized: now 1790435600062457 us; error 61 us; skew 500 ppm
I20260926 15:13:20.062834 29240 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:13:20 dist-test-slave-865v krb5kdc[27919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435600, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:13:20.066437 29240 init.cc:377] Logged in from keytab as kudu/127.12.34.194@KRBTEST.COM (short username kudu)
I20260926 15:13:20.068473 29240 webserver.cc:460] Webserver started at http://127.12.34.194:36799/ using document root <none> and password file <none>
I20260926 15:13:20.069280 29240 fs_manager.cc:359] Metadata directory not provided
I20260926 15:13:20.069438 29240 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:13:20.075596 29240 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 15:13:20.078517 29255 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:13:20.080184 29240 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 15:13:20.080396 29240 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/ts-1/data,/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/ts-1/wal
uuid: "9fda66f1b152460e8e0cb199f028a5c0"
format_stamp: "Formatted at 2026-09-26 15:12:34 on dist-test-slave-865v"
server_key: "fd39586b7675ea818ec3df58e8cc92a9"
server_key_iv: "d6367cc0dc316147b92055a8d8fed163"
server_key_version: "encryptionkey@0"
I20260926 15:13:20.081537 29240 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:13:20.105942 29240 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:13:20.111439 29240 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:13:20.111763 29240 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:13:20.112812 29240 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:13:20.114293 29240 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:13:20.114430 29240 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:13:20.114568 29240 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:13:20.114658 29240 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:13:20.153673 29240 rpc_server.cc:307] RPC server started. Bound to: 127.12.34.194:38857
I20260926 15:13:20.153766 29367 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.34.194:38857 every 8 connection(s)
I20260926 15:13:20.155112 29240 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/ts-1/data/info.pb
I20260926 15:13:20.160168 12427 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8uoJGv/build/asan/bin/kudu as pid 29240
I20260926 15:13:20.160351 29370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:20.158268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:45601 (local address 127.12.34.194:53727)
0926 15:13:20.159045 (+ 777us) negotiation.cc:107] Waiting for socket to connect
0926 15:13:20.159305 (+ 260us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.254:45601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":309,"threads_started":1}
I20260926 15:13:20.160971 12427 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8uoJGv/build/asan/bin/kudu
/tmp/dist-test-task8uoJGv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.34.195:44121
--local_ip_for_outbound_sockets=127.12.34.195
--tserver_master_addrs=127.12.34.254:45601
--webserver_port=33713
--webserver_interface=127.12.34.195
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.195.keytab
--principal=kudu/127.12.34.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.34.212:40893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:13:20.161010 29368 heartbeater.cc:644] Failed to heartbeat to 127.12.34.254:45601 (0 consecutive failures): Network error: Failed to ping master at 127.12.34.254:45601: Client connection negotiation failed: client connection to 127.12.34.254:45601: connect: Connection refused (error 111)
I20260926 15:13:20.162031 29370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:20.161568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:45601 (local address 127.12.34.194:53511)
0926 15:13:20.161776 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0926 15:13:20.161872 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.254:45601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260926 15:13:20.163079 29370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:20.162624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:45601 (local address 127.12.34.194:33329)
0926 15:13:20.162794 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0926 15:13:20.162896 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.254:45601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260926 15:13:20.163439 29368 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 15:13:20.493275 29371 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:13:20.493531 29371 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:13:20.493625 29371 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:13:20.500674 29371 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:13:20.500771 29371 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:13:20.500993 29371 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.34.195
I20260926 15:13:20.510351 29371 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.34.212:40893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.195.keytab
--principal=kudu/127.12.34.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.34.195:44121
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.34.195
--webserver_port=33713
--tserver_master_addrs=127.12.34.254:45601
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.34.195
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:13:20.511621 29371 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:13:20.512928 29371 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:13:20.526970 29379 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:13:20.526969 29377 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:13:20.527256 29376 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:13:20.528262 29371 server_base.cc:1034] running on GCE node
I20260926 15:13:20.528919 29371 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:13:20.529942 29371 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:13:20.531211 29371 hybrid_clock.cc:648] HybridClock initialized: now 1790435600531162 us; error 67 us; skew 500 ppm
I20260926 15:13:20.531565 29371 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:13:20 dist-test-slave-865v krb5kdc[27919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435600, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:13:20.534998 29371 init.cc:377] Logged in from keytab as kudu/127.12.34.195@KRBTEST.COM (short username kudu)
I20260926 15:13:20.536369 29371 webserver.cc:460] Webserver started at http://127.12.34.195:33713/ using document root <none> and password file <none>
I20260926 15:13:20.537017 29371 fs_manager.cc:359] Metadata directory not provided
I20260926 15:13:20.537186 29371 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:13:20.542577 29371 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260926 15:13:20.545243 29386 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:13:20.546573 29371 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 15:13:20.546759 29371 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/ts-2/data,/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/ts-2/wal
uuid: "6e3b1a3f8e1343239a873c89195df725"
format_stamp: "Formatted at 2026-09-26 15:12:35 on dist-test-slave-865v"
server_key: "5f8da085443f5841cf5692d98bb00261"
server_key_iv: "d68a194c90806156ba2e288e7d369e21"
server_key_version: "encryptionkey@0"
I20260926 15:13:20.547685 29371 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:13:20.563422 29371 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:13:20.568428 29371 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:13:20.568786 29371 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:13:20.570109 29371 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:13:20.571820 29371 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:13:20.571946 29371 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:13:20.572072 29371 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:13:20.572173 29371 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:13:20.603624 29371 rpc_server.cc:307] RPC server started. Bound to: 127.12.34.195:44121
I20260926 15:13:20.603667 29498 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.34.195:44121 every 8 connection(s)
I20260926 15:13:20.604997 29371 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790435318278681-12427-0/minicluster-data/ts-2/data/info.pb
I20260926 15:13:20.608022 12427 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8uoJGv/build/asan/bin/kudu as pid 29371
I20260926 15:13:20.609900 29501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:20.607871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:45601 (local address 127.12.34.195:37701)
0926 15:13:20.608674 (+ 803us) negotiation.cc:107] Waiting for socket to connect
0926 15:13:20.608959 (+ 285us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.254:45601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":353,"threads_started":1}
W20260926 15:13:20.610522 29499 heartbeater.cc:644] Failed to heartbeat to 127.12.34.254:45601 (0 consecutive failures): Network error: Failed to ping master at 127.12.34.254:45601: Client connection negotiation failed: client connection to 127.12.34.254:45601: connect: Connection refused (error 111)
I20260926 15:13:20.611248 29501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:20.610957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:45601 (local address 127.12.34.195:56869)
0926 15:13:20.611086 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0926 15:13:20.611155 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.254:45601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260926 15:13:20.612041 29501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:20.611750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:45601 (local address 127.12.34.195:42197)
0926 15:13:20.611877 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0926 15:13:20.611946 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.254:45601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260926 15:13:20.612332 29499 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 15:13:20.713404 29503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:20.712460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:45601 (local address 127.12.34.193:41315)
0926 15:13:20.712929 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0926 15:13:20.713011 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.254:45601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":236,"threads_started":1}
I20260926 15:13:21.164875 29504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:21.164069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:45601 (local address 127.12.34.194:48309)
0926 15:13:21.164528 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0926 15:13:21.164618 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.254:45601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":219,"threads_started":1}
I20260926 15:13:21.613883 29505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:21.612887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:45601 (local address 127.12.34.195:47715)
0926 15:13:21.613506 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0926 15:13:21.613626 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.254:45601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":331,"threads_started":1}
I20260926 15:13:21.715206 29506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:21.714473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:45601 (local address 127.12.34.193:56419)
0926 15:13:21.714962 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0926 15:13:21.715052 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.254:45601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":259,"threads_started":1}
I20260926 15:13:22.166803 29507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:22.166017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:45601 (local address 127.12.34.194:49723)
0926 15:13:22.166516 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0926 15:13:22.166606 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.254:45601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":248,"threads_started":1}
I20260926 15:13:22.615541 29508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:22.614842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:45601 (local address 127.12.34.195:55007)
0926 15:13:22.615304 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0926 15:13:22.615387 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.254:45601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":215,"threads_started":1}
I20260926 15:13:22.717139 29509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:22.716345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:45601 (local address 127.12.34.193:49585)
0926 15:13:22.716846 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0926 15:13:22.716939 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.254:45601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":244,"threads_started":1}
I20260926 15:13:23.168473 29510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:23.167769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:45601 (local address 127.12.34.194:40695)
0926 15:13:23.168207 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0926 15:13:23.168302 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.254:45601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":238,"threads_started":1}
I20260926 15:13:23.617756 29511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:23.616573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:45601 (local address 127.12.34.195:46475)
0926 15:13:23.617423 (+ 850us) negotiation.cc:107] Waiting for socket to connect
0926 15:13:23.617527 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.254:45601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":726,"thread_start_us":350,"threads_started":1}
I20260926 15:13:23.718935 29512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:23.718131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:45601 (local address 127.12.34.193:57685)
0926 15:13:23.718614 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0926 15:13:23.718703 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.254:45601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":267,"threads_started":1}
I20260926 15:13:24.170156 29513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:24.169451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:45601 (local address 127.12.34.194:38179)
0926 15:13:24.169904 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0926 15:13:24.169987 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.254:45601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":231,"threads_started":1}
I20260926 15:13:24.619499 29514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:24.618800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:45601 (local address 127.12.34.195:44875)
0926 15:13:24.619240 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0926 15:13:24.619316 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.254:45601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":230,"threads_started":1}
I20260926 15:13:24.721231 29515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:24.720303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:45601 (local address 127.12.34.193:42285)
0926 15:13:24.720872 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0926 15:13:24.720967 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.254:45601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":350,"threads_started":1}
I20260926 15:13:25.171919 29516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:25.171117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:45601 (local address 127.12.34.194:53313)
0926 15:13:25.171581 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0926 15:13:25.171658 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.254:45601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":252,"threads_started":1}
I20260926 15:13:25.621162 29517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:25.620365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:45601 (local address 127.12.34.195:54579)
0926 15:13:25.620850 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0926 15:13:25.620930 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.254:45601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":235,"threads_started":1}
I20260926 15:13:25.722750 29518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:25.722098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:45601 (local address 127.12.34.193:36153)
0926 15:13:25.722520 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0926 15:13:25.722597 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.254:45601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":229,"threads_started":1}
I20260926 15:13:26.173666 29519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:26.172887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:45601 (local address 127.12.34.194:34815)
0926 15:13:26.173415 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0926 15:13:26.173504 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.254:45601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":280,"threads_started":1}
I20260926 15:13:26.622671 29520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:26.621927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:45601 (local address 127.12.34.195:47091)
0926 15:13:26.622379 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0926 15:13:26.622468 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.254:45601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":229,"threads_started":1}
I20260926 15:13:26.724532 29521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:26.723668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:45601 (local address 127.12.34.193:38627)
0926 15:13:26.724235 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0926 15:13:26.724318 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.254:45601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":308,"threads_started":1}
I20260926 15:13:27.175650 29522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:27.174773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:45601 (local address 127.12.34.194:43085)
0926 15:13:27.175362 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0926 15:13:27.175441 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.254:45601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":275,"threads_started":1}
I20260926 15:13:27.624436 29523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:27.623604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:45601 (local address 127.12.34.195:54707)
0926 15:13:27.624150 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0926 15:13:27.624235 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.254:45601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":284,"threads_started":1}
I20260926 15:13:27.726002 29524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:27.725379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:45601 (local address 127.12.34.193:44135)
0926 15:13:27.725757 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0926 15:13:27.725832 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.254:45601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":200,"threads_started":1}
I20260926 15:13:28.177348 29525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:28.176520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:45601 (local address 127.12.34.194:45353)
0926 15:13:28.177043 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0926 15:13:28.177167 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.254:45601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":287,"threads_started":1}
I20260926 15:13:28.626078 29526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:28.625373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:45601 (local address 127.12.34.195:35495)
0926 15:13:28.625823 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0926 15:13:28.625903 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.254:45601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":235,"threads_started":1}
I20260926 15:13:28.727843 29527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:28.726975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:45601 (local address 127.12.34.193:53937)
0926 15:13:28.727525 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0926 15:13:28.727631 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.254:45601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":274,"threads_started":1}
I20260926 15:13:29.179034 29528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:29.178345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:45601 (local address 127.12.34.194:33587)
0926 15:13:29.178790 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0926 15:13:29.178868 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.254:45601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":209,"threads_started":1}
I20260926 15:13:29.627833 29529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:29.626968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:45601 (local address 127.12.34.195:44471)
0926 15:13:29.627543 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0926 15:13:29.627617 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.254:45601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":288,"threads_started":1}
I20260926 15:13:29.729527 29530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:29.728773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:45601 (local address 127.12.34.193:57595)
0926 15:13:29.729278 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0926 15:13:29.729356 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.254:45601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":261,"threads_started":1}
I20260926 15:13:30.181526 29531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:30.180270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:45601 (local address 127.12.34.194:37387)
0926 15:13:30.181143 (+ 873us) negotiation.cc:107] Waiting for socket to connect
0926 15:13:30.181277 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.254:45601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":669,"thread_start_us":485,"threads_started":1}
I20260926 15:13:30.629459 29532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:30.628699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:45601 (local address 127.12.34.195:39065)
0926 15:13:30.629220 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0926 15:13:30.629296 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.254:45601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":260,"threads_started":1}
I20260926 15:13:30.731222 29533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:30.730422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:45601 (local address 127.12.34.193:53311)
0926 15:13:30.730907 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0926 15:13:30.730992 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.254:45601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":269,"threads_started":1}
I20260926 15:13:31.183184 29534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:31.182476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:45601 (local address 127.12.34.194:33395)
0926 15:13:31.182917 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0926 15:13:31.183009 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.254:45601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":199,"threads_started":1}
I20260926 15:13:31.631096 29535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:31.630405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:45601 (local address 127.12.34.195:58857)
0926 15:13:31.630831 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0926 15:13:31.630927 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.254:45601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":212,"threads_started":1}
I20260926 15:13:31.732923 29536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:31.732204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:45601 (local address 127.12.34.193:51201)
0926 15:13:31.732642 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0926 15:13:31.732737 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.254:45601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":243,"threads_started":1}
I20260926 15:13:32.185215 29537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:32.184164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:45601 (local address 127.12.34.194:56831)
0926 15:13:32.184752 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0926 15:13:32.184910 (+ 158us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.254:45601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":322,"threads_started":1}
I20260926 15:13:32.632619 29538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:32.631957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:45601 (local address 127.12.34.195:52689)
0926 15:13:32.632386 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0926 15:13:32.632468 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.254:45601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":223,"threads_started":1}
I20260926 15:13:32.734639 29539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:32.733891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:45601 (local address 127.12.34.193:57559)
0926 15:13:32.734360 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0926 15:13:32.734457 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.254:45601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":246,"threads_started":1}
I20260926 15:13:33.187117 29540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:33.186361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:45601 (local address 127.12.34.194:45461)
0926 15:13:33.186826 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0926 15:13:33.186931 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.254:45601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":210,"threads_started":1}
I20260926 15:13:33.634383 29541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:33.633605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:45601 (local address 127.12.34.195:51467)
0926 15:13:33.634110 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0926 15:13:33.634203 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.254:45601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":254,"threads_started":1}
I20260926 15:13:33.736303 29542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:33.735564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:45601 (local address 127.12.34.193:58345)
0926 15:13:33.736047 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0926 15:13:33.736129 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.254:45601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":238,"threads_started":1}
I20260926 15:13:34.188659 29543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:34.187931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:45601 (local address 127.12.34.194:39639)
0926 15:13:34.188400 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0926 15:13:34.188483 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.254:45601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":250,"threads_started":1}
I20260926 15:13:34.635943 29544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:34.635258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:45601 (local address 127.12.34.195:42411)
0926 15:13:34.635692 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0926 15:13:34.635770 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.254:45601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":209,"threads_started":1}
I20260926 15:13:34.739050 29545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:34.737746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:45601 (local address 127.12.34.193:34883)
0926 15:13:34.738635 (+ 889us) negotiation.cc:107] Waiting for socket to connect
0926 15:13:34.738797 (+ 162us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.254:45601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":734,"thread_start_us":463,"threads_started":1}
I20260926 15:13:35.190454 29546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:35.189746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:45601 (local address 127.12.34.194:55149)
0926 15:13:35.190159 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0926 15:13:35.190275 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.254:45601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":220,"threads_started":1}
I20260926 15:13:35.612941 12427 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 29068
I20260926 15:13:35.631048 29547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:35.614096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59724 (local address 127.12.34.193:45901)
0926 15:13:35.614665 (+ 569us) server_negotiation.cc:206] Beginning negotiation
0926 15:13:35.614749 (+ 84us) server_negotiation.cc:399] Waiting for connection header
0926 15:13:35.614824 (+ 75us) server_negotiation.cc:407] Connection header received
0926 15:13:35.616905 (+ 2081us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:13:35.616914 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:13:35.617416 (+ 502us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:13:35.617704 (+ 288us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:13:35.619074 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:35.620270 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:35.621567 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:35.622215 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:35.622344 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:13:35.625203 (+ 2859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:13:35.625211 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:13:35.625225 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:13:35.625263 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:13:35.628092 (+ 2829us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:13:35.628762 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:13:35.628767 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:13:35.628773 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:13:35.628926 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:13:35.629327 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:13:35.629330 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:13:35.629332 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:13:35.629802 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:13:35.630015 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0926 15:13:35.630371 (+ 356us) server_negotiation.cc:299] Negotiation successful
0926 15:13:35.630752 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":404,"thread_start_us":286,"threads_started":1}
I20260926 15:13:35.632476 29232 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:13:35.637841 29549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:35.637192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:45601 (local address 127.12.34.195:55443)
0926 15:13:35.637586 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0926 15:13:35.637667 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.254:45601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":198,"threads_started":1}
I20260926 15:13:35.718236 12427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8uoJGv/build/asan/bin/kudu with pid 29068
I20260926 15:13:35.736002 12427 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 29240
I20260926 15:13:35.753437 29550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:35.736928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40130 (local address 127.12.34.194:38857)
0926 15:13:35.737433 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0926 15:13:35.737446 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 15:13:35.737481 (+ 35us) server_negotiation.cc:407] Connection header received
0926 15:13:35.738905 (+ 1424us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:13:35.738914 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:13:35.739321 (+ 407us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:13:35.739654 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:13:35.741070 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:35.742351 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:35.743580 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:35.744242 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:35.744429 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:13:35.747180 (+ 2751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:13:35.747188 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:13:35.747202 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:13:35.747243 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:13:35.750179 (+ 2936us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:13:35.750860 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:13:35.750867 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:13:35.750873 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:13:35.751110 (+ 237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:13:35.751461 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:13:35.751467 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:13:35.751470 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:13:35.751948 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:13:35.752258 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0926 15:13:35.752698 (+ 440us) server_negotiation.cc:299] Negotiation successful
0926 15:13:35.753134 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":256,"threads_started":1}
I20260926 15:13:35.754765 29363 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:13:35.837508 12427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8uoJGv/build/asan/bin/kudu with pid 29240
I20260926 15:13:35.854339 12427 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 29371
I20260926 15:13:35.871690 29552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:35.855299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58176 (local address 127.12.34.195:44121)
0926 15:13:35.855889 (+ 590us) server_negotiation.cc:206] Beginning negotiation
0926 15:13:35.855903 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 15:13:35.855942 (+ 39us) server_negotiation.cc:407] Connection header received
0926 15:13:35.857701 (+ 1759us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:13:35.857711 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:13:35.858121 (+ 410us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:13:35.858390 (+ 269us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:13:35.859660 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:35.860834 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:35.862130 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:35.862769 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:35.862892 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:13:35.865558 (+ 2666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:13:35.865566 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:13:35.865580 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:13:35.865625 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:13:35.868577 (+ 2952us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:13:35.869270 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:13:35.869277 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:13:35.869284 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:13:35.869438 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:13:35.869816 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:13:35.869820 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:13:35.869822 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:13:35.870304 (+ 482us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:13:35.870605 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0926 15:13:35.871007 (+ 402us) server_negotiation.cc:299] Negotiation successful
0926 15:13:35.871396 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":442,"thread_start_us":250,"threads_started":1}
I20260926 15:13:35.872973 29494 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:13:35.955582 12427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8uoJGv/build/asan/bin/kudu with pid 29371
2026-09-26T15:13:35Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (64113 ms)
[----------] 30 tests from SecurityITest (297640 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435318278681-12427-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 15:13:36 dist-test-slave-865v krb5kdc[29556](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 15:13:36 dist-test-slave-865v krb5kdc[29556](info): set up 2 sockets
Sep 26 15:13:36 dist-test-slave-865v krb5kdc[29556](info): commencing operation
krb5kdc: starting...
W20260926 15:13:37.073236 12427 mini_kdc.cc:121] Time spent starting KDC: real 1.078s user 0.002s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435318278681-12427-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435318278681-12427-0/krb5kdc/test-user.keytab.
Sep 26 15:13:37 dist-test-slave-865v krb5kdc[29556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435617, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T15:13:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T15:13:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.34.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.34.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.34.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.254.keytab.
Entry for principal kudu/127.12.34.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.254.keytab.
I20260926 15:13:37.314327 12427 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8uoJGv/build/asan/bin/kudu
/tmp/dist-test-task8uoJGv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435318278681-12427-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435318278681-12427-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435318278681-12427-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.34.254:36297
--webserver_interface=127.12.34.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.254.keytab
--principal=kudu/127.12.34.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.34.212:44805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.34.254:36297
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435318278681-12427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435318278681-12427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435318278681-12427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:13:37.637861 29569 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:13:37.638176 29569 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:13:37.638324 29569 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:13:37.645507 29569 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:13:37.645637 29569 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:13:37.645735 29569 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:13:37.645828 29569 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:13:37.645898 29569 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:13:37.655459 29569 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.34.212:44805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435318278681-12427-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435318278681-12427-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.34.254:36297
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.254.keytab
--principal=kudu/127.12.34.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.34.254:36297
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.34.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435318278681-12427-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:13:37.656806 29569 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:13:37.658118 29569 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:13:37.669512 29574 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:13:37.669608 29577 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:13:37.669637 29575 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:13:37.670665 29569 server_base.cc:1034] running on GCE node
I20260926 15:13:37.671360 29569 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:13:37.672678 29569 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:13:37.673903 29569 hybrid_clock.cc:648] HybridClock initialized: now 1790435617673854 us; error 52 us; skew 500 ppm
I20260926 15:13:37.674239 29569 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:13:37 dist-test-slave-865v krb5kdc[29556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435617, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:13:37.682739 29569 init.cc:377] Logged in from keytab as kudu/127.12.34.254@KRBTEST.COM (short username kudu)
I20260926 15:13:37.684415 29569 webserver.cc:460] Webserver started at http://127.12.34.254:45791/ using document root <none> and password file <none>
I20260926 15:13:37.685179 29569 fs_manager.cc:359] Metadata directory not provided
I20260926 15:13:37.685297 29569 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:13:37.685549 29569 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:13:37.688431 29569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435318278681-12427-0/minicluster-data/master-0/data/instance:
uuid: "79ccf4a6ac4343b29a95c22795f142b5"
format_stamp: "Formatted at 2026-09-26 15:13:37 on dist-test-slave-865v"
server_key: "89346c1e964c117d61f42c060eeb993c"
server_key_iv: "55bda250b356cf52ca4cfeb168f65711"
server_key_version: "encryptionkey@0"
I20260926 15:13:37.689183 29569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435318278681-12427-0/minicluster-data/master-0/wal/instance:
uuid: "79ccf4a6ac4343b29a95c22795f142b5"
format_stamp: "Formatted at 2026-09-26 15:13:37 on dist-test-slave-865v"
server_key: "89346c1e964c117d61f42c060eeb993c"
server_key_iv: "55bda250b356cf52ca4cfeb168f65711"
server_key_version: "encryptionkey@0"
I20260926 15:13:37.694195 29569 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260926 15:13:37.697531 29584 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:13:37.698913 29569 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 15:13:37.699123 29569 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435318278681-12427-0/minicluster-data/master-0/data,/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435318278681-12427-0/minicluster-data/master-0/wal
uuid: "79ccf4a6ac4343b29a95c22795f142b5"
format_stamp: "Formatted at 2026-09-26 15:13:37 on dist-test-slave-865v"
server_key: "89346c1e964c117d61f42c060eeb993c"
server_key_iv: "55bda250b356cf52ca4cfeb168f65711"
server_key_version: "encryptionkey@0"
I20260926 15:13:37.699365 29569 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435318278681-12427-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435318278681-12427-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435318278681-12427-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:13:37.713573 29569 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:13:37.717710 29569 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:13:37.718027 29569 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:13:37.736084 29569 rpc_server.cc:307] RPC server started. Bound to: 127.12.34.254:36297
I20260926 15:13:37.736107 29635 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.34.254:36297 every 8 connection(s)
I20260926 15:13:37.737473 29569 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
I20260926 15:13:37.737905 12427 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8uoJGv/build/asan/bin/kudu as pid 29569
I20260926 15:13:37.738101 12427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435318278681-12427-0/minicluster-data/master-0/wal/instance
I20260926 15:13:37.738901 12427 external_mini_cluster.cc:1428] Setting key a31e4634bc663b574bde062c24c1b316
I20260926 15:13:37.741621 29636 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 26 15:13:37 dist-test-slave-865v krb5kdc[29556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435617, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.34.254@KRBTEST.COM
I20260926 15:13:37.754674 29636 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79ccf4a6ac4343b29a95c22795f142b5: Bootstrap starting.
I20260926 15:13:37.757817 29636 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 79ccf4a6ac4343b29a95c22795f142b5: Neither blocks nor log segments found. Creating new log.
I20260926 15:13:37.758857 29636 log.cc:824] T 00000000000000000000000000000000 P 79ccf4a6ac4343b29a95c22795f142b5: Log is configured to *not* fsync() on all Append() calls
I20260926 15:13:37.761678 29636 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79ccf4a6ac4343b29a95c22795f142b5: No bootstrap required, opened a new log
I20260926 15:13:37.761946 29639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:37.741050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54946 (local address 127.12.34.254:36297)
0926 15:13:37.741927 (+ 877us) server_negotiation.cc:206] Beginning negotiation
0926 15:13:37.741938 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 15:13:37.741982 (+ 44us) server_negotiation.cc:407] Connection header received
0926 15:13:37.743558 (+ 1576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:13:37.743578 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:13:37.743949 (+ 371us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:13:37.744265 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:13:37.746108 (+ 1843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:37.747463 (+ 1355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:37.749343 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:37.750050 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:37.750186 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:13:37.754962 (+ 4776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:13:37.754970 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:13:37.754983 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:13:37.755020 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:13:37.758077 (+ 3057us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:13:37.758786 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:13:37.758792 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:13:37.758798 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:13:37.758971 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:13:37.759356 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:13:37.759366 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:13:37.759369 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:13:37.759886 (+ 517us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:13:37.760186 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0926 15:13:37.760558 (+ 372us) server_negotiation.cc:299] Negotiation successful
0926 15:13:37.760954 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":660,"thread_start_us":376,"threads_started":1}
I20260926 15:13:37.769466 29636 raft_consensus.cc:348] T 00000000000000000000000000000000 P 79ccf4a6ac4343b29a95c22795f142b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79ccf4a6ac4343b29a95c22795f142b5" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 36297 } }
I20260926 15:13:37.769806 29636 raft_consensus.cc:374] T 00000000000000000000000000000000 P 79ccf4a6ac4343b29a95c22795f142b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:13:37.769908 29636 raft_consensus.cc:729] T 00000000000000000000000000000000 P 79ccf4a6ac4343b29a95c22795f142b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79ccf4a6ac4343b29a95c22795f142b5, State: Initialized, Role: FOLLOWER
I20260926 15:13:37.770483 29636 consensus_queue.cc:260] T 00000000000000000000000000000000 P 79ccf4a6ac4343b29a95c22795f142b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79ccf4a6ac4343b29a95c22795f142b5" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 36297 } }
I20260926 15:13:37.770747 29636 raft_consensus.cc:388] T 00000000000000000000000000000000 P 79ccf4a6ac4343b29a95c22795f142b5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 15:13:37.770944 29636 raft_consensus.cc:482] T 00000000000000000000000000000000 P 79ccf4a6ac4343b29a95c22795f142b5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 15:13:37.771126 29636 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 79ccf4a6ac4343b29a95c22795f142b5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:13:37.773144 29636 raft_consensus.cc:504] T 00000000000000000000000000000000 P 79ccf4a6ac4343b29a95c22795f142b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79ccf4a6ac4343b29a95c22795f142b5" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 36297 } }
I20260926 15:13:37.773562 29636 leader_election.cc:302] T 00000000000000000000000000000000 P 79ccf4a6ac4343b29a95c22795f142b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 79ccf4a6ac4343b29a95c22795f142b5; no voters:
I20260926 15:13:37.774099 29636 leader_election.cc:288] T 00000000000000000000000000000000 P 79ccf4a6ac4343b29a95c22795f142b5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 15:13:37.774259 29641 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 79ccf4a6ac4343b29a95c22795f142b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:13:37.774999 29641 raft_consensus.cc:686] T 00000000000000000000000000000000 P 79ccf4a6ac4343b29a95c22795f142b5 [term 1 LEADER]: Becoming Leader. State: Replica: 79ccf4a6ac4343b29a95c22795f142b5, State: Running, Role: LEADER
I20260926 15:13:37.775538 29641 consensus_queue.cc:237] T 00000000000000000000000000000000 P 79ccf4a6ac4343b29a95c22795f142b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79ccf4a6ac4343b29a95c22795f142b5" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 36297 } }
I20260926 15:13:37.776355 29636 sys_catalog.cc:564] T 00000000000000000000000000000000 P 79ccf4a6ac4343b29a95c22795f142b5 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:13:37.778294 29642 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79ccf4a6ac4343b29a95c22795f142b5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "79ccf4a6ac4343b29a95c22795f142b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79ccf4a6ac4343b29a95c22795f142b5" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 36297 } } }
I20260926 15:13:37.778566 29643 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79ccf4a6ac4343b29a95c22795f142b5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 79ccf4a6ac4343b29a95c22795f142b5. Latest consensus state: current_term: 1 leader_uuid: "79ccf4a6ac4343b29a95c22795f142b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79ccf4a6ac4343b29a95c22795f142b5" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 36297 } } }
I20260926 15:13:37.778770 29642 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79ccf4a6ac4343b29a95c22795f142b5 [sys.catalog]: This master's current role is: LEADER
I20260926 15:13:37.778859 29643 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79ccf4a6ac4343b29a95c22795f142b5 [sys.catalog]: This master's current role is: LEADER
I20260926 15:13:37.780812 29649 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:13:37.785899 29649 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 15:13:37.794337 29649 catalog_manager.cc:1347] Generated new cluster ID: 703a4641c26d4709a373bd8ff6902c40
I20260926 15:13:37.794440 29649 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 15:13:37.812474 29649 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 15:13:37.813762 29649 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:13:37.828984 29649 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 79ccf4a6ac4343b29a95c22795f142b5: Generated new TSK 0
I20260926 15:13:37.830083 29649 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.34.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.34.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.34.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.193.keytab.
Entry for principal kudu/127.12.34.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.193.keytab.
I20260926 15:13:37.913671 12427 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8uoJGv/build/asan/bin/kudu
/tmp/dist-test-task8uoJGv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435318278681-12427-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435318278681-12427-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435318278681-12427-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435318278681-12427-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.34.193:0
--local_ip_for_outbound_sockets=127.12.34.193
--webserver_interface=127.12.34.193
--webserver_port=0
--tserver_master_addrs=127.12.34.254:36297
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.193.keytab
--principal=kudu/127.12.34.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.34.212:44805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435318278681-12427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435318278681-12427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435318278681-12427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:13:38.235972 29662 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:13:38.236281 29662 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:13:38.236451 29662 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:13:38.243685 29662 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:13:38.243826 29662 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:13:38.244004 29662 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.34.193
I20260926 15:13:38.253518 29662 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.34.212:44805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435318278681-12427-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435318278681-12427-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.193.keytab
--principal=kudu/127.12.34.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.34.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435318278681-12427-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.34.193
--webserver_port=0
--tserver_master_addrs=127.12.34.254:36297
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.34.193
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435318278681-12427-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:13:38.254690 29662 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:13:38.255896 29662 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:13:38.268504 29667 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:13:38.268435 29668 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:13:38.268641 29670 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:13:38.269407 29662 server_base.cc:1034] running on GCE node
I20260926 15:13:38.269946 29662 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:13:38.270799 29662 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:13:38.272030 29662 hybrid_clock.cc:648] HybridClock initialized: now 1790435618271988 us; error 41 us; skew 500 ppm
I20260926 15:13:38.272329 29662 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:13:38 dist-test-slave-865v krb5kdc[29556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435618, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:13:38.275332 29662 init.cc:377] Logged in from keytab as kudu/127.12.34.193@KRBTEST.COM (short username kudu)
I20260926 15:13:38.276732 29662 webserver.cc:460] Webserver started at http://127.12.34.193:44181/ using document root <none> and password file <none>
I20260926 15:13:38.277475 29662 fs_manager.cc:359] Metadata directory not provided
I20260926 15:13:38.277607 29662 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:13:38.277940 29662 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:13:38.280787 29662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435318278681-12427-0/minicluster-data/ts-0/data/instance:
uuid: "5e67688cfdb54d55b398e907387884a2"
format_stamp: "Formatted at 2026-09-26 15:13:38 on dist-test-slave-865v"
server_key: "8e539d661d1c4f5a0b32447113c09cd4"
server_key_iv: "c4d5fe753c5d76842db4f970956e1028"
server_key_version: "encryptionkey@0"
I20260926 15:13:38.281530 29662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435318278681-12427-0/minicluster-data/ts-0/wal/instance:
uuid: "5e67688cfdb54d55b398e907387884a2"
format_stamp: "Formatted at 2026-09-26 15:13:38 on dist-test-slave-865v"
server_key: "8e539d661d1c4f5a0b32447113c09cd4"
server_key_iv: "c4d5fe753c5d76842db4f970956e1028"
server_key_version: "encryptionkey@0"
I20260926 15:13:38.286314 29662 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 15:13:38.289871 29677 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:13:38.291132 29662 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 15:13:38.291313 29662 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435318278681-12427-0/minicluster-data/ts-0/data,/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435318278681-12427-0/minicluster-data/ts-0/wal
uuid: "5e67688cfdb54d55b398e907387884a2"
format_stamp: "Formatted at 2026-09-26 15:13:38 on dist-test-slave-865v"
server_key: "8e539d661d1c4f5a0b32447113c09cd4"
server_key_iv: "c4d5fe753c5d76842db4f970956e1028"
server_key_version: "encryptionkey@0"
I20260926 15:13:38.291515 29662 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435318278681-12427-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435318278681-12427-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435318278681-12427-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:13:38.306334 29662 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:13:38.310189 29662 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:13:38.310498 29662 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:13:38.311412 29662 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:13:38.312696 29662 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:13:38.312803 29662 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:13:38.312945 29662 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:13:38.313020 29662 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:13:38.345659 29662 rpc_server.cc:307] RPC server started. Bound to: 127.12.34.193:37247
I20260926 15:13:38.345711 29789 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.34.193:37247 every 8 connection(s)
I20260926 15:13:38.346922 29662 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435318278681-12427-0/minicluster-data/ts-0/data/info.pb
I20260926 15:13:38.349332 12427 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8uoJGv/build/asan/bin/kudu as pid 29662
I20260926 15:13:38.349547 12427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435318278681-12427-0/minicluster-data/ts-0/wal/instance
I20260926 15:13:38.350347 12427 external_mini_cluster.cc:1428] Setting key a479b74c3736657021186e5b39eab6fe
Sep 26 15:13:38 dist-test-slave-865v krb5kdc[29556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435618, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.193@KRBTEST.COM for kudu/127.12.34.254@KRBTEST.COM
I20260926 15:13:38.366515 29792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:38.349362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.193:38259 (local address 127.12.34.254:36297)
0926 15:13:38.349813 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0926 15:13:38.349819 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:13:38.350960 (+ 1141us) server_negotiation.cc:407] Connection header received
0926 15:13:38.352719 (+ 1759us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:13:38.352724 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:13:38.352812 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:13:38.352926 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:13:38.354373 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:38.355057 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:38.356536 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:38.356928 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:38.357057 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:13:38.360645 (+ 3588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:13:38.360649 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:13:38.360653 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:13:38.360684 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:13:38.363243 (+ 2559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:13:38.364169 (+ 926us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:13:38.364173 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:13:38.364175 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:13:38.364302 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:13:38.364768 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:13:38.364772 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:13:38.364775 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:13:38.365029 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:13:38.365237 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0926 15:13:38.366080 (+ 843us) server_negotiation.cc:299] Negotiation successful
0926 15:13:38.366250 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":236,"threads_started":1}
I20260926 15:13:38.368088 29793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:38.350036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:36297 (local address 127.12.34.193:38259)
0926 15:13:38.350804 (+ 768us) negotiation.cc:107] Waiting for socket to connect
0926 15:13:38.350847 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 15:13:38.352464 (+ 1617us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:13:38.353188 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:13:38.353196 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:13:38.353686 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:13:38.354206 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:13:38.354221 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:13:38.355217 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:13:38.355221 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:13:38.356396 (+ 1175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:13:38.356408 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:13:38.357200 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:13:38.357209 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:13:38.357428 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:13:38.358089 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:13:38.358110 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:13:38.360433 (+ 2323us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:13:38.363391 (+ 2958us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:13:38.363403 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:13:38.363422 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:13:38.364052 (+ 630us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:13:38.364405 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:13:38.364413 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:13:38.364420 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:13:38.364661 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:13:38.365221 (+ 560us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:13:38.365232 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:13:38.365666 (+ 434us) client_negotiation.cc:769] Sending connection context
0926 15:13:38.365972 (+ 306us) client_negotiation.cc:240] Negotiation successful
0926 15:13:38.366543 (+ 571us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":337,"threads_started":1}
I20260926 15:13:38.369715 29790 heartbeater.cc:347] Connected to a master server at 127.12.34.254:36297
I20260926 15:13:38.370048 29790 heartbeater.cc:464] Registering TS with master...
I20260926 15:13:38.370837 29790 heartbeater.cc:511] Master 127.12.34.254:36297 requested a full tablet report, sending...
I20260926 15:13:38.372895 29601 ts_manager.cc:194] Registered new tserver with Master: 5e67688cfdb54d55b398e907387884a2 (127.12.34.193:37247)
WARNING: no policy specified for kudu/127.12.34.194@KRBTEST.COM; defaulting to no policy
I20260926 15:13:38.374346 29601 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.34.193@KRBTEST.COM'} at 127.12.34.193:38259
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.34.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.34.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.194.keytab.
Entry for principal kudu/127.12.34.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.194.keytab.
I20260926 15:13:38.404556 12427 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8uoJGv/build/asan/bin/kudu
/tmp/dist-test-task8uoJGv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435318278681-12427-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435318278681-12427-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435318278681-12427-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435318278681-12427-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.34.194:0
--local_ip_for_outbound_sockets=127.12.34.194
--webserver_interface=127.12.34.194
--webserver_port=0
--tserver_master_addrs=127.12.34.254:36297
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.194.keytab
--principal=kudu/127.12.34.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.34.212:44805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435318278681-12427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435318278681-12427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435318278681-12427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:13:38.724411 29796 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:13:38.724675 29796 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:13:38.724803 29796 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:13:38.731859 29796 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:13:38.731956 29796 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:13:38.732060 29796 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.34.194
I20260926 15:13:38.745621 29796 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.34.212:44805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435318278681-12427-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435318278681-12427-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.194.keytab
--principal=kudu/127.12.34.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.34.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435318278681-12427-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.34.194
--webserver_port=0
--tserver_master_addrs=127.12.34.254:36297
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.34.194
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435318278681-12427-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:13:38.747104 29796 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:13:38.748588 29796 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:13:38.762740 29801 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:13:38.762688 29802 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:13:38.762665 29804 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:13:38.763420 29796 server_base.cc:1034] running on GCE node
I20260926 15:13:38.763935 29796 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:13:38.764850 29796 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:13:38.766043 29796 hybrid_clock.cc:648] HybridClock initialized: now 1790435618766000 us; error 37 us; skew 500 ppm
I20260926 15:13:38.766341 29796 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:13:38 dist-test-slave-865v krb5kdc[29556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435618, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:13:38.769462 29796 init.cc:377] Logged in from keytab as kudu/127.12.34.194@KRBTEST.COM (short username kudu)
I20260926 15:13:38.770985 29796 webserver.cc:460] Webserver started at http://127.12.34.194:45911/ using document root <none> and password file <none>
I20260926 15:13:38.772017 29796 fs_manager.cc:359] Metadata directory not provided
I20260926 15:13:38.772167 29796 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:13:38.772493 29796 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:13:38.775377 29796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435318278681-12427-0/minicluster-data/ts-1/data/instance:
uuid: "d090c183506248a3884279bf1c50dc6a"
format_stamp: "Formatted at 2026-09-26 15:13:38 on dist-test-slave-865v"
server_key: "404e309000cce489412a075aed67205a"
server_key_iv: "f5c4e434df2cf7cb597b0f65bc2f443c"
server_key_version: "encryptionkey@0"
I20260926 15:13:38.776117 29796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435318278681-12427-0/minicluster-data/ts-1/wal/instance:
uuid: "d090c183506248a3884279bf1c50dc6a"
format_stamp: "Formatted at 2026-09-26 15:13:38 on dist-test-slave-865v"
server_key: "404e309000cce489412a075aed67205a"
server_key_iv: "f5c4e434df2cf7cb597b0f65bc2f443c"
server_key_version: "encryptionkey@0"
I20260926 15:13:38.781064 29796 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260926 15:13:38.784499 29811 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:13:38.785924 29796 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 15:13:38.786090 29796 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435318278681-12427-0/minicluster-data/ts-1/data,/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435318278681-12427-0/minicluster-data/ts-1/wal
uuid: "d090c183506248a3884279bf1c50dc6a"
format_stamp: "Formatted at 2026-09-26 15:13:38 on dist-test-slave-865v"
server_key: "404e309000cce489412a075aed67205a"
server_key_iv: "f5c4e434df2cf7cb597b0f65bc2f443c"
server_key_version: "encryptionkey@0"
I20260926 15:13:38.786288 29796 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435318278681-12427-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435318278681-12427-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435318278681-12427-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:13:38.832139 29796 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:13:38.836185 29796 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:13:38.836463 29796 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:13:38.837440 29796 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:13:38.838699 29796 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:13:38.838814 29796 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:13:38.838958 29796 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:13:38.839033 29796 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:13:38.872020 29796 rpc_server.cc:307] RPC server started. Bound to: 127.12.34.194:34895
I20260926 15:13:38.872068 29923 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.34.194:34895 every 8 connection(s)
I20260926 15:13:38.873334 29796 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435318278681-12427-0/minicluster-data/ts-1/data/info.pb
I20260926 15:13:38.877542 12427 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8uoJGv/build/asan/bin/kudu as pid 29796
I20260926 15:13:38.877750 12427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435318278681-12427-0/minicluster-data/ts-1/wal/instance
I20260926 15:13:38.878175 12427 external_mini_cluster.cc:1428] Setting key 6a641aba2ae6cea36b002d70c74d0a70
Sep 26 15:13:38 dist-test-slave-865v krb5kdc[29556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435618, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.194@KRBTEST.COM for kudu/127.12.34.254@KRBTEST.COM
I20260926 15:13:38.894857 29926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:38.875761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.194:43929 (local address 127.12.34.254:36297)
0926 15:13:38.876268 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0926 15:13:38.876272 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:13:38.877321 (+ 1049us) server_negotiation.cc:407] Connection header received
0926 15:13:38.879446 (+ 2125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:13:38.879450 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:13:38.879519 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:13:38.879607 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:13:38.881371 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:38.882611 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:38.884094 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:38.884841 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:38.884974 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:13:38.888357 (+ 3383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:13:38.888367 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:13:38.888372 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:13:38.888418 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:13:38.891513 (+ 3095us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:13:38.892200 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:13:38.892206 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:13:38.892209 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:13:38.892339 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:13:38.892776 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:13:38.892784 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:13:38.892786 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:13:38.893037 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:13:38.893238 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0926 15:13:38.894011 (+ 773us) server_negotiation.cc:299] Negotiation successful
0926 15:13:38.894399 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":246,"threads_started":1}
I20260926 15:13:38.895834 29927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:38.876333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:36297 (local address 127.12.34.194:43929)
0926 15:13:38.877102 (+ 769us) negotiation.cc:107] Waiting for socket to connect
0926 15:13:38.877171 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0926 15:13:38.879143 (+ 1972us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:13:38.879863 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:13:38.879874 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:13:38.880599 (+ 725us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:13:38.881201 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:13:38.881219 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:13:38.882781 (+ 1562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:13:38.882794 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:13:38.883951 (+ 1157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:13:38.883962 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:13:38.885040 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:13:38.885045 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:13:38.885303 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:13:38.886000 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:13:38.886022 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:13:38.888138 (+ 2116us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:13:38.891647 (+ 3509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:13:38.891656 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:13:38.891672 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:13:38.892078 (+ 406us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:13:38.892452 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:13:38.892457 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:13:38.892461 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:13:38.892656 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:13:38.893208 (+ 552us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:13:38.893216 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:13:38.893538 (+ 322us) client_negotiation.cc:769] Sending connection context
0926 15:13:38.893864 (+ 326us) client_negotiation.cc:240] Negotiation successful
0926 15:13:38.894382 (+ 518us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":324,"threads_started":1}
I20260926 15:13:38.897271 29924 heartbeater.cc:347] Connected to a master server at 127.12.34.254:36297
I20260926 15:13:38.897583 29924 heartbeater.cc:464] Registering TS with master...
I20260926 15:13:38.898433 29924 heartbeater.cc:511] Master 127.12.34.254:36297 requested a full tablet report, sending...
I20260926 15:13:38.899935 29601 ts_manager.cc:194] Registered new tserver with Master: d090c183506248a3884279bf1c50dc6a (127.12.34.194:34895)
I20260926 15:13:38.900714 29601 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.34.194@KRBTEST.COM'} at 127.12.34.194:43929
WARNING: no policy specified for kudu/127.12.34.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.34.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.34.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.195.keytab.
Entry for principal kudu/127.12.34.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.195.keytab.
I20260926 15:13:38.932559 12427 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8uoJGv/build/asan/bin/kudu
/tmp/dist-test-task8uoJGv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435318278681-12427-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435318278681-12427-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435318278681-12427-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435318278681-12427-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.34.195:0
--local_ip_for_outbound_sockets=127.12.34.195
--webserver_interface=127.12.34.195
--webserver_port=0
--tserver_master_addrs=127.12.34.254:36297
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.195.keytab
--principal=kudu/127.12.34.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.34.212:44805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435318278681-12427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435318278681-12427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435318278681-12427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:13:39.247247 29930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:13:39.247501 29930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:13:39.247597 29930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:13:39.254621 29930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:13:39.254734 29930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:13:39.254845 29930 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.34.195
I20260926 15:13:39.264087 29930 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.34.212:44805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435318278681-12427-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435318278681-12427-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.195.keytab
--principal=kudu/127.12.34.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.34.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435318278681-12427-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.34.195
--webserver_port=0
--tserver_master_addrs=127.12.34.254:36297
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.34.195
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435318278681-12427-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:13:39.265215 29930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:13:39.266395 29930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:13:39.279542 29938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:13:39.279614 29935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:13:39.279590 29936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:13:39.280144 29930 server_base.cc:1034] running on GCE node
I20260926 15:13:39.280802 29930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:13:39.281785 29930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:13:39.282976 29930 hybrid_clock.cc:648] HybridClock initialized: now 1790435619282923 us; error 47 us; skew 500 ppm
I20260926 15:13:39.283291 29930 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:13:39 dist-test-slave-865v krb5kdc[29556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435619, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:13:39.286475 29930 init.cc:377] Logged in from keytab as kudu/127.12.34.195@KRBTEST.COM (short username kudu)
I20260926 15:13:39.288015 29930 webserver.cc:460] Webserver started at http://127.12.34.195:45885/ using document root <none> and password file <none>
I20260926 15:13:39.288659 29930 fs_manager.cc:359] Metadata directory not provided
I20260926 15:13:39.288796 29930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:13:39.289137 29930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:13:39.292311 29930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435318278681-12427-0/minicluster-data/ts-2/data/instance:
uuid: "c1dfcec90ece48028dc7121315b1492a"
format_stamp: "Formatted at 2026-09-26 15:13:39 on dist-test-slave-865v"
server_key: "64fc8c9db15822d1cf91add122063d33"
server_key_iv: "b917d7d0f547d06d00065ab7ab5988ad"
server_key_version: "encryptionkey@0"
I20260926 15:13:39.293147 29930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435318278681-12427-0/minicluster-data/ts-2/wal/instance:
uuid: "c1dfcec90ece48028dc7121315b1492a"
format_stamp: "Formatted at 2026-09-26 15:13:39 on dist-test-slave-865v"
server_key: "64fc8c9db15822d1cf91add122063d33"
server_key_iv: "b917d7d0f547d06d00065ab7ab5988ad"
server_key_version: "encryptionkey@0"
I20260926 15:13:39.298333 29930 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260926 15:13:39.302129 29945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:13:39.303525 29930 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 15:13:39.303661 29930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435318278681-12427-0/minicluster-data/ts-2/data,/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435318278681-12427-0/minicluster-data/ts-2/wal
uuid: "c1dfcec90ece48028dc7121315b1492a"
format_stamp: "Formatted at 2026-09-26 15:13:39 on dist-test-slave-865v"
server_key: "64fc8c9db15822d1cf91add122063d33"
server_key_iv: "b917d7d0f547d06d00065ab7ab5988ad"
server_key_version: "encryptionkey@0"
I20260926 15:13:39.303849 29930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435318278681-12427-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435318278681-12427-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435318278681-12427-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:13:39.322373 29930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:13:39.326299 29930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:13:39.326576 29930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:13:39.327545 29930 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:13:39.328836 29930 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:13:39.328910 29930 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:13:39.329061 29930 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:13:39.329149 29930 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260926 15:13:39.362494 29930 rpc_server.cc:307] RPC server started. Bound to: 127.12.34.195:41157
I20260926 15:13:39.362576 30057 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.34.195:41157 every 8 connection(s)
I20260926 15:13:39.363686 29930 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435318278681-12427-0/minicluster-data/ts-2/data/info.pb
I20260926 15:13:39.365531 12427 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8uoJGv/build/asan/bin/kudu as pid 29930
I20260926 15:13:39.365710 12427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435318278681-12427-0/minicluster-data/ts-2/wal/instance
I20260926 15:13:39.366271 12427 external_mini_cluster.cc:1428] Setting key 4ed6a6b79b7208fbe5bb87fb082c1719
I20260926 15:13:39.377357 29790 heartbeater.cc:503] Master 127.12.34.254:36297 was elected leader, sending a full tablet report...
Sep 26 15:13:39 dist-test-slave-865v krb5kdc[29556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435619, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.195@KRBTEST.COM for kudu/127.12.34.254@KRBTEST.COM
I20260926 15:13:39.384934 29926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:39.366401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.195:51793 (local address 127.12.34.254:36297)
0926 15:13:39.366570 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0926 15:13:39.366575 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:13:39.367883 (+ 1308us) server_negotiation.cc:407] Connection header received
0926 15:13:39.370104 (+ 2221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:13:39.370107 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:13:39.370188 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:13:39.370275 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:13:39.371603 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:39.372353 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:39.374020 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:39.374425 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:39.374528 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:13:39.378640 (+ 4112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:13:39.378646 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:13:39.378652 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:13:39.378697 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:13:39.382061 (+ 3364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:13:39.382770 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:13:39.382775 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:13:39.382779 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:13:39.382901 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:13:39.383311 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:13:39.383316 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:13:39.383319 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:13:39.383588 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:13:39.383752 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0926 15:13:39.384397 (+ 645us) server_negotiation.cc:299] Negotiation successful
0926 15:13:39.384614 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 15:13:39.385840 30060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:39.366948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:36297 (local address 127.12.34.195:51793)
0926 15:13:39.367710 (+ 762us) negotiation.cc:107] Waiting for socket to connect
0926 15:13:39.367780 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0926 15:13:39.369847 (+ 2067us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:13:39.370429 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:13:39.370436 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:13:39.370939 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:13:39.371443 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:13:39.371480 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:13:39.372510 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:13:39.372514 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:13:39.373884 (+ 1370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:13:39.373896 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:13:39.374631 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:13:39.374669 (+ 38us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:13:39.374851 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:13:39.375486 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:13:39.375527 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:13:39.378453 (+ 2926us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:13:39.382211 (+ 3758us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:13:39.382220 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:13:39.382235 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:13:39.382645 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:13:39.383006 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:13:39.383012 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:13:39.383015 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:13:39.383197 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:13:39.383729 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:13:39.383738 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:13:39.384051 (+ 313us) client_negotiation.cc:769] Sending connection context
0926 15:13:39.384300 (+ 249us) client_negotiation.cc:240] Negotiation successful
0926 15:13:39.384798 (+ 498us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":289,"threads_started":1}
I20260926 15:13:39.387086 30058 heartbeater.cc:347] Connected to a master server at 127.12.34.254:36297
I20260926 15:13:39.387324 30058 heartbeater.cc:464] Registering TS with master...
I20260926 15:13:39.387971 30058 heartbeater.cc:511] Master 127.12.34.254:36297 requested a full tablet report, sending...
I20260926 15:13:39.389283 29601 ts_manager.cc:194] Registered new tserver with Master: c1dfcec90ece48028dc7121315b1492a (127.12.34.195:41157)
I20260926 15:13:39.390254 29601 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.34.195@KRBTEST.COM'} at 127.12.34.195:51793
I20260926 15:13:39.393795 12427 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 15:13:39.410270 29926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:39.398157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59344 (local address 127.12.34.254:36297)
0926 15:13:39.398369 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0926 15:13:39.398378 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 15:13:39.398695 (+ 317us) server_negotiation.cc:407] Connection header received
0926 15:13:39.398815 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:13:39.398818 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:13:39.398880 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:13:39.398958 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:13:39.400326 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:39.401057 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:39.402396 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:39.402922 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:39.403017 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:13:39.405402 (+ 2385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:13:39.405407 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:13:39.405411 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:13:39.405440 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:13:39.408005 (+ 2565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:13:39.408563 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:13:39.408566 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:13:39.408569 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:13:39.408676 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:13:39.408993 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:13:39.408996 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:13:39.408998 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:13:39.409361 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:13:39.409545 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0926 15:13:39.409803 (+ 258us) server_negotiation.cc:299] Negotiation successful
0926 15:13:39.409951 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 15:13:39.413884 29601 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59344:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 15:13:39.420938 29601 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 15:13:39.443560 30072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:39.433415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35890 (local address 127.12.34.195:41157)
0926 15:13:39.435252 (+ 1837us) server_negotiation.cc:206] Beginning negotiation
0926 15:13:39.435261 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 15:13:39.435295 (+ 34us) server_negotiation.cc:407] Connection header received
0926 15:13:39.435393 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:13:39.435400 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:13:39.435650 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:13:39.436707 (+ 1057us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:13:39.436986 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:39.438475 (+ 1489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:39.441516 (+ 3041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:39.442440 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:39.442593 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:13:39.442854 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0926 15:13:39.443043 (+ 189us) server_negotiation.cc:299] Negotiation successful
0926 15:13:39.443205 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1616,"thread_start_us":2093,"threads_started":1}
I20260926 15:13:39.443672 30070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:39.432977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.195:41157 (local address 127.0.0.1:35890)
0926 15:13:39.433743 (+ 766us) negotiation.cc:107] Waiting for socket to connect
0926 15:13:39.433785 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 15:13:39.433949 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:13:39.435901 (+ 1952us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:13:39.435912 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:13:39.435954 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:13:39.436486 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:13:39.436501 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:13:39.438593 (+ 2092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:13:39.438601 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:13:39.441352 (+ 2751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:13:39.441367 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:13:39.442618 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:13:39.442624 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:13:39.442801 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:13:39.442858 (+ 57us) client_negotiation.cc:769] Sending connection context
0926 15:13:39.443005 (+ 147us) client_negotiation.cc:240] Negotiation successful
0926 15:13:39.443205 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"mutex_wait_us":76,"thread_start_us":383,"threads_started":1}
I20260926 15:13:39.444368 30075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:39.434975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.194:34895 (local address 127.0.0.1:56076)
0926 15:13:39.436223 (+ 1248us) negotiation.cc:107] Waiting for socket to connect
0926 15:13:39.436244 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 15:13:39.436389 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:13:39.437700 (+ 1311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:13:39.437706 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:13:39.437730 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:13:39.438046 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:13:39.438056 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:13:39.439459 (+ 1403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:13:39.439464 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:13:39.441483 (+ 2019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:13:39.441498 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:13:39.443479 (+ 1981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:13:39.443484 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:13:39.443684 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:13:39.443748 (+ 64us) client_negotiation.cc:769] Sending connection context
0926 15:13:39.443824 (+ 76us) client_negotiation.cc:240] Negotiation successful
0926 15:13:39.443940 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1056,"mutex_wait_us":495,"thread_start_us":381,"threads_started":1}
I20260926 15:13:39.444612 30074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:39.435330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56076 (local address 127.12.34.194:34895)
0926 15:13:39.436941 (+ 1611us) server_negotiation.cc:206] Beginning negotiation
0926 15:13:39.436950 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 15:13:39.436973 (+ 23us) server_negotiation.cc:407] Connection header received
0926 15:13:39.437059 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:13:39.437068 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:13:39.437314 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:13:39.437497 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:13:39.438213 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:39.439321 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:39.441628 (+ 2307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:39.442982 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:39.443108 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:13:39.443278 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0926 15:13:39.444186 (+ 908us) server_negotiation.cc:299] Negotiation successful
0926 15:13:39.444341 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1374,"thread_start_us":1268,"threads_started":1}
I20260926 15:13:39.445022 30073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:39.434075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40552 (local address 127.12.34.193:37247)
0926 15:13:39.435481 (+ 1406us) server_negotiation.cc:206] Beginning negotiation
0926 15:13:39.435490 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 15:13:39.435518 (+ 28us) server_negotiation.cc:407] Connection header received
0926 15:13:39.435625 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:13:39.435635 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:13:39.435880 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:13:39.436082 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:13:39.437137 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:39.438566 (+ 1429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:39.441554 (+ 2988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:39.442965 (+ 1411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:39.443105 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:13:39.443308 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0926 15:13:39.444582 (+ 1274us) server_negotiation.cc:299] Negotiation successful
0926 15:13:39.444760 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1221,"thread_start_us":1248,"threads_started":1}
I20260926 15:13:39.445472 30071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:39.433297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.193:37247 (local address 127.0.0.1:40552)
0926 15:13:39.435114 (+ 1817us) negotiation.cc:107] Waiting for socket to connect
0926 15:13:39.435131 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 15:13:39.435243 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:13:39.436427 (+ 1184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:13:39.436435 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:13:39.436475 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:13:39.436894 (+ 419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:13:39.436907 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:13:39.438676 (+ 1769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:13:39.438682 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:13:39.441359 (+ 2677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:13:39.441367 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:13:39.443465 (+ 2098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:13:39.443472 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:13:39.443692 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:13:39.443972 (+ 280us) client_negotiation.cc:769] Sending connection context
0926 15:13:39.444050 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 15:13:39.444174 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1690,"mutex_wait_us":391,"thread_start_us":1517,"threads_started":1}
I20260926 15:13:39.447715 29993 tablet_service.cc:1467] Processing CreateTablet for tablet 95e4d2bfb49346d091f236eb8b565193 (DEFAULT_TABLE table=test-table [id=0ccfae505cc8409bb82c6d074af259f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:13:39.448390 29859 tablet_service.cc:1467] Processing CreateTablet for tablet 95e4d2bfb49346d091f236eb8b565193 (DEFAULT_TABLE table=test-table [id=0ccfae505cc8409bb82c6d074af259f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:13:39.448968 29993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95e4d2bfb49346d091f236eb8b565193. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:13:39.448676 29725 tablet_service.cc:1467] Processing CreateTablet for tablet 95e4d2bfb49346d091f236eb8b565193 (DEFAULT_TABLE table=test-table [id=0ccfae505cc8409bb82c6d074af259f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:13:39.449694 29859 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95e4d2bfb49346d091f236eb8b565193. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:13:39.449909 29725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95e4d2bfb49346d091f236eb8b565193. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:13:39.459986 30076 tablet_bootstrap.cc:492] T 95e4d2bfb49346d091f236eb8b565193 P c1dfcec90ece48028dc7121315b1492a: Bootstrap starting.
I20260926 15:13:39.463218 30076 tablet_bootstrap.cc:654] T 95e4d2bfb49346d091f236eb8b565193 P c1dfcec90ece48028dc7121315b1492a: Neither blocks nor log segments found. Creating new log.
I20260926 15:13:39.465026 30078 tablet_bootstrap.cc:492] T 95e4d2bfb49346d091f236eb8b565193 P 5e67688cfdb54d55b398e907387884a2: Bootstrap starting.
I20260926 15:13:39.465461 30077 tablet_bootstrap.cc:492] T 95e4d2bfb49346d091f236eb8b565193 P d090c183506248a3884279bf1c50dc6a: Bootstrap starting.
I20260926 15:13:39.465479 30076 log.cc:824] T 95e4d2bfb49346d091f236eb8b565193 P c1dfcec90ece48028dc7121315b1492a: Log is configured to *not* fsync() on all Append() calls
I20260926 15:13:39.468271 30078 tablet_bootstrap.cc:654] T 95e4d2bfb49346d091f236eb8b565193 P 5e67688cfdb54d55b398e907387884a2: Neither blocks nor log segments found. Creating new log.
I20260926 15:13:39.468484 30077 tablet_bootstrap.cc:654] T 95e4d2bfb49346d091f236eb8b565193 P d090c183506248a3884279bf1c50dc6a: Neither blocks nor log segments found. Creating new log.
I20260926 15:13:39.468664 30076 tablet_bootstrap.cc:492] T 95e4d2bfb49346d091f236eb8b565193 P c1dfcec90ece48028dc7121315b1492a: No bootstrap required, opened a new log
I20260926 15:13:39.468926 30076 ts_tablet_manager.cc:1397] T 95e4d2bfb49346d091f236eb8b565193 P c1dfcec90ece48028dc7121315b1492a: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20260926 15:13:39.469487 30077 log.cc:824] T 95e4d2bfb49346d091f236eb8b565193 P d090c183506248a3884279bf1c50dc6a: Log is configured to *not* fsync() on all Append() calls
I20260926 15:13:39.469470 30078 log.cc:824] T 95e4d2bfb49346d091f236eb8b565193 P 5e67688cfdb54d55b398e907387884a2: Log is configured to *not* fsync() on all Append() calls
I20260926 15:13:39.471725 30077 tablet_bootstrap.cc:492] T 95e4d2bfb49346d091f236eb8b565193 P d090c183506248a3884279bf1c50dc6a: No bootstrap required, opened a new log
I20260926 15:13:39.471959 30077 ts_tablet_manager.cc:1397] T 95e4d2bfb49346d091f236eb8b565193 P d090c183506248a3884279bf1c50dc6a: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260926 15:13:39.478494 30078 tablet_bootstrap.cc:492] T 95e4d2bfb49346d091f236eb8b565193 P 5e67688cfdb54d55b398e907387884a2: No bootstrap required, opened a new log
I20260926 15:13:39.478755 30078 ts_tablet_manager.cc:1397] T 95e4d2bfb49346d091f236eb8b565193 P 5e67688cfdb54d55b398e907387884a2: Time spent bootstrapping tablet: real 0.014s user 0.011s sys 0.000s
I20260926 15:13:39.479372 30077 raft_consensus.cc:348] T 95e4d2bfb49346d091f236eb8b565193 P d090c183506248a3884279bf1c50dc6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1dfcec90ece48028dc7121315b1492a" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 41157 } } peers { permanent_uuid: "5e67688cfdb54d55b398e907387884a2" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 37247 } } peers { permanent_uuid: "d090c183506248a3884279bf1c50dc6a" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 34895 } }
I20260926 15:13:39.479700 30077 raft_consensus.cc:374] T 95e4d2bfb49346d091f236eb8b565193 P d090c183506248a3884279bf1c50dc6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:13:39.479804 30077 raft_consensus.cc:729] T 95e4d2bfb49346d091f236eb8b565193 P d090c183506248a3884279bf1c50dc6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d090c183506248a3884279bf1c50dc6a, State: Initialized, Role: FOLLOWER
I20260926 15:13:39.480412 30077 consensus_queue.cc:260] T 95e4d2bfb49346d091f236eb8b565193 P d090c183506248a3884279bf1c50dc6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1dfcec90ece48028dc7121315b1492a" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 41157 } } peers { permanent_uuid: "5e67688cfdb54d55b398e907387884a2" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 37247 } } peers { permanent_uuid: "d090c183506248a3884279bf1c50dc6a" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 34895 } }
I20260926 15:13:39.481563 29924 heartbeater.cc:503] Master 127.12.34.254:36297 was elected leader, sending a full tablet report...
I20260926 15:13:39.481619 30076 raft_consensus.cc:348] T 95e4d2bfb49346d091f236eb8b565193 P c1dfcec90ece48028dc7121315b1492a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1dfcec90ece48028dc7121315b1492a" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 41157 } } peers { permanent_uuid: "5e67688cfdb54d55b398e907387884a2" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 37247 } } peers { permanent_uuid: "d090c183506248a3884279bf1c50dc6a" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 34895 } }
I20260926 15:13:39.482026 30076 raft_consensus.cc:374] T 95e4d2bfb49346d091f236eb8b565193 P c1dfcec90ece48028dc7121315b1492a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:13:39.482126 30076 raft_consensus.cc:729] T 95e4d2bfb49346d091f236eb8b565193 P c1dfcec90ece48028dc7121315b1492a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1dfcec90ece48028dc7121315b1492a, State: Initialized, Role: FOLLOWER
I20260926 15:13:39.483475 30077 ts_tablet_manager.cc:1428] T 95e4d2bfb49346d091f236eb8b565193 P d090c183506248a3884279bf1c50dc6a: Time spent starting tablet: real 0.011s user 0.001s sys 0.010s
I20260926 15:13:39.482697 30076 consensus_queue.cc:260] T 95e4d2bfb49346d091f236eb8b565193 P c1dfcec90ece48028dc7121315b1492a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1dfcec90ece48028dc7121315b1492a" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 41157 } } peers { permanent_uuid: "5e67688cfdb54d55b398e907387884a2" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 37247 } } peers { permanent_uuid: "d090c183506248a3884279bf1c50dc6a" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 34895 } }
I20260926 15:13:39.484828 30058 heartbeater.cc:503] Master 127.12.34.254:36297 was elected leader, sending a full tablet report...
I20260926 15:13:39.486697 30076 ts_tablet_manager.cc:1428] T 95e4d2bfb49346d091f236eb8b565193 P c1dfcec90ece48028dc7121315b1492a: Time spent starting tablet: real 0.018s user 0.017s sys 0.000s
I20260926 15:13:39.488736 30078 raft_consensus.cc:348] T 95e4d2bfb49346d091f236eb8b565193 P 5e67688cfdb54d55b398e907387884a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1dfcec90ece48028dc7121315b1492a" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 41157 } } peers { permanent_uuid: "5e67688cfdb54d55b398e907387884a2" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 37247 } } peers { permanent_uuid: "d090c183506248a3884279bf1c50dc6a" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 34895 } }
I20260926 15:13:39.489065 30078 raft_consensus.cc:374] T 95e4d2bfb49346d091f236eb8b565193 P 5e67688cfdb54d55b398e907387884a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:13:39.489219 30078 raft_consensus.cc:729] T 95e4d2bfb49346d091f236eb8b565193 P 5e67688cfdb54d55b398e907387884a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e67688cfdb54d55b398e907387884a2, State: Initialized, Role: FOLLOWER
I20260926 15:13:39.489713 30078 consensus_queue.cc:260] T 95e4d2bfb49346d091f236eb8b565193 P 5e67688cfdb54d55b398e907387884a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1dfcec90ece48028dc7121315b1492a" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 41157 } } peers { permanent_uuid: "5e67688cfdb54d55b398e907387884a2" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 37247 } } peers { permanent_uuid: "d090c183506248a3884279bf1c50dc6a" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 34895 } }
I20260926 15:13:39.491194 30078 ts_tablet_manager.cc:1428] T 95e4d2bfb49346d091f236eb8b565193 P 5e67688cfdb54d55b398e907387884a2: Time spent starting tablet: real 0.012s user 0.013s sys 0.000s
I20260926 15:13:39.512600 30084 raft_consensus.cc:482] T 95e4d2bfb49346d091f236eb8b565193 P 5e67688cfdb54d55b398e907387884a2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 15:13:39.512954 30084 raft_consensus.cc:504] T 95e4d2bfb49346d091f236eb8b565193 P 5e67688cfdb54d55b398e907387884a2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1dfcec90ece48028dc7121315b1492a" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 41157 } } peers { permanent_uuid: "5e67688cfdb54d55b398e907387884a2" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 37247 } } peers { permanent_uuid: "d090c183506248a3884279bf1c50dc6a" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 34895 } }
I20260926 15:13:39.514569 30084 leader_election.cc:288] T 95e4d2bfb49346d091f236eb8b565193 P 5e67688cfdb54d55b398e907387884a2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c1dfcec90ece48028dc7121315b1492a (127.12.34.195:41157), d090c183506248a3884279bf1c50dc6a (127.12.34.194:34895)
I20260926 15:13:39.523464 30072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:39.515129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.193:34849 (local address 127.12.34.195:41157)
0926 15:13:39.515351 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0926 15:13:39.515361 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 15:13:39.515933 (+ 572us) server_negotiation.cc:407] Connection header received
0926 15:13:39.516079 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:13:39.516084 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:13:39.516151 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:13:39.516246 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:13:39.517137 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:39.518257 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:39.521154 (+ 2897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:39.522388 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:39.522551 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:13:39.522624 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 15:13:39.523072 (+ 448us) server_negotiation.cc:299] Negotiation successful
0926 15:13:39.523200 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 15:13:39.523454 30074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:39.515132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.193:34353 (local address 127.12.34.194:34895)
0926 15:13:39.515373 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0926 15:13:39.515379 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:13:39.515830 (+ 451us) server_negotiation.cc:407] Connection header received
0926 15:13:39.516055 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:13:39.516061 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:13:39.516140 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:13:39.516246 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:13:39.517137 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:39.518240 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:39.521160 (+ 2920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:39.522391 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:39.522543 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:13:39.522615 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 15:13:39.523072 (+ 457us) server_negotiation.cc:299] Negotiation successful
0926 15:13:39.523213 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260926 15:13:39.523701 30085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:39.514952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.194:34895 (local address 127.12.34.193:34353)
0926 15:13:39.515714 (+ 762us) negotiation.cc:107] Waiting for socket to connect
0926 15:13:39.515737 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 15:13:39.515893 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:13:39.516303 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:13:39.516308 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:13:39.516339 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:13:39.516946 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:13:39.516955 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:13:39.518405 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:13:39.518409 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:13:39.520992 (+ 2583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:13:39.521003 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:13:39.522633 (+ 1630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:13:39.522641 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:13:39.522884 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:13:39.522933 (+ 49us) client_negotiation.cc:769] Sending connection context
0926 15:13:39.523026 (+ 93us) client_negotiation.cc:240] Negotiation successful
0926 15:13:39.523254 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":588,"mutex_wait_us":364,"thread_start_us":422,"threads_started":1}
I20260926 15:13:39.523835 30086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:39.515000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.195:41157 (local address 127.12.34.193:34849)
0926 15:13:39.515843 (+ 843us) negotiation.cc:107] Waiting for socket to connect
0926 15:13:39.515857 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 15:13:39.515959 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:13:39.516303 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:13:39.516308 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:13:39.516391 (+ 83us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:13:39.516921 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:13:39.516941 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:13:39.518399 (+ 1458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:13:39.518405 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:13:39.520988 (+ 2583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:13:39.521003 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:13:39.522649 (+ 1646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:13:39.522653 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:13:39.522880 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:13:39.522931 (+ 51us) client_negotiation.cc:769] Sending connection context
0926 15:13:39.523026 (+ 95us) client_negotiation.cc:240] Negotiation successful
0926 15:13:39.523254 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":735,"mutex_wait_us":219,"spinlock_wait_cycles":23296,"thread_start_us":559,"threads_started":1}
I20260926 15:13:39.524703 30013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95e4d2bfb49346d091f236eb8b565193" candidate_uuid: "5e67688cfdb54d55b398e907387884a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1dfcec90ece48028dc7121315b1492a" is_pre_election: true
I20260926 15:13:39.524727 29879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95e4d2bfb49346d091f236eb8b565193" candidate_uuid: "5e67688cfdb54d55b398e907387884a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d090c183506248a3884279bf1c50dc6a" is_pre_election: true
I20260926 15:13:39.525203 29879 raft_consensus.cc:2454] T 95e4d2bfb49346d091f236eb8b565193 P d090c183506248a3884279bf1c50dc6a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e67688cfdb54d55b398e907387884a2 in term 0.
I20260926 15:13:39.525202 30013 raft_consensus.cc:2454] T 95e4d2bfb49346d091f236eb8b565193 P c1dfcec90ece48028dc7121315b1492a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e67688cfdb54d55b398e907387884a2 in term 0.
I20260926 15:13:39.525811 29679 leader_election.cc:302] T 95e4d2bfb49346d091f236eb8b565193 P 5e67688cfdb54d55b398e907387884a2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 5e67688cfdb54d55b398e907387884a2, c1dfcec90ece48028dc7121315b1492a, d090c183506248a3884279bf1c50dc6a; no voters:
I20260926 15:13:39.526156 30084 raft_consensus.cc:2781] T 95e4d2bfb49346d091f236eb8b565193 P 5e67688cfdb54d55b398e907387884a2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 15:13:39.526295 30084 raft_consensus.cc:482] T 95e4d2bfb49346d091f236eb8b565193 P 5e67688cfdb54d55b398e907387884a2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 15:13:39.526400 30084 raft_consensus.cc:3037] T 95e4d2bfb49346d091f236eb8b565193 P 5e67688cfdb54d55b398e907387884a2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:13:39.528519 30084 raft_consensus.cc:504] T 95e4d2bfb49346d091f236eb8b565193 P 5e67688cfdb54d55b398e907387884a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1dfcec90ece48028dc7121315b1492a" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 41157 } } peers { permanent_uuid: "5e67688cfdb54d55b398e907387884a2" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 37247 } } peers { permanent_uuid: "d090c183506248a3884279bf1c50dc6a" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 34895 } }
I20260926 15:13:39.529135 30084 leader_election.cc:288] T 95e4d2bfb49346d091f236eb8b565193 P 5e67688cfdb54d55b398e907387884a2 [CANDIDATE]: Term 1 election: Requested vote from peers c1dfcec90ece48028dc7121315b1492a (127.12.34.195:41157), d090c183506248a3884279bf1c50dc6a (127.12.34.194:34895)
I20260926 15:13:39.529511 30013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95e4d2bfb49346d091f236eb8b565193" candidate_uuid: "5e67688cfdb54d55b398e907387884a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1dfcec90ece48028dc7121315b1492a"
I20260926 15:13:39.529533 29879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95e4d2bfb49346d091f236eb8b565193" candidate_uuid: "5e67688cfdb54d55b398e907387884a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d090c183506248a3884279bf1c50dc6a"
I20260926 15:13:39.529695 30013 raft_consensus.cc:3037] T 95e4d2bfb49346d091f236eb8b565193 P c1dfcec90ece48028dc7121315b1492a [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:13:39.529706 29879 raft_consensus.cc:3037] T 95e4d2bfb49346d091f236eb8b565193 P d090c183506248a3884279bf1c50dc6a [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:13:39.531868 30013 raft_consensus.cc:2454] T 95e4d2bfb49346d091f236eb8b565193 P c1dfcec90ece48028dc7121315b1492a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e67688cfdb54d55b398e907387884a2 in term 1.
I20260926 15:13:39.532029 29879 raft_consensus.cc:2454] T 95e4d2bfb49346d091f236eb8b565193 P d090c183506248a3884279bf1c50dc6a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e67688cfdb54d55b398e907387884a2 in term 1.
I20260926 15:13:39.532208 29679 leader_election.cc:302] T 95e4d2bfb49346d091f236eb8b565193 P 5e67688cfdb54d55b398e907387884a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e67688cfdb54d55b398e907387884a2, c1dfcec90ece48028dc7121315b1492a; no voters:
I20260926 15:13:39.532505 30084 raft_consensus.cc:2781] T 95e4d2bfb49346d091f236eb8b565193 P 5e67688cfdb54d55b398e907387884a2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:13:39.532946 30084 raft_consensus.cc:686] T 95e4d2bfb49346d091f236eb8b565193 P 5e67688cfdb54d55b398e907387884a2 [term 1 LEADER]: Becoming Leader. State: Replica: 5e67688cfdb54d55b398e907387884a2, State: Running, Role: LEADER
I20260926 15:13:39.533437 30084 consensus_queue.cc:237] T 95e4d2bfb49346d091f236eb8b565193 P 5e67688cfdb54d55b398e907387884a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1dfcec90ece48028dc7121315b1492a" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 41157 } } peers { permanent_uuid: "5e67688cfdb54d55b398e907387884a2" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 37247 } } peers { permanent_uuid: "d090c183506248a3884279bf1c50dc6a" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 34895 } }
I20260926 15:13:39.537748 29600 catalog_manager.cc:5495] T 95e4d2bfb49346d091f236eb8b565193 P 5e67688cfdb54d55b398e907387884a2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5e67688cfdb54d55b398e907387884a2 (127.12.34.193). New cstate: current_term: 1 leader_uuid: "5e67688cfdb54d55b398e907387884a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1dfcec90ece48028dc7121315b1492a" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 41157 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e67688cfdb54d55b398e907387884a2" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 37247 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d090c183506248a3884279bf1c50dc6a" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 34895 } health_report { overall_health: UNKNOWN } } }
I20260926 15:13:39.567981 30073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:39.562971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40554 (local address 127.12.34.193:37247)
0926 15:13:39.563146 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0926 15:13:39.563151 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:13:39.563167 (+ 16us) server_negotiation.cc:407] Connection header received
0926 15:13:39.563245 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:13:39.563249 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:13:39.563337 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:13:39.563455 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 15:13:39.563755 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:39.564468 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:39.565864 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:39.566304 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:39.566402 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:13:39.566896 (+ 494us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 15:13:39.567427 (+ 531us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 15:13:39.567541 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0926 15:13:39.567705 (+ 164us) server_negotiation.cc:299] Negotiation successful
0926 15:13:39.567769 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 15:13:39.575979 30013 raft_consensus.cc:1261] T 95e4d2bfb49346d091f236eb8b565193 P c1dfcec90ece48028dc7121315b1492a [term 1 FOLLOWER]: Refusing update from remote peer 5e67688cfdb54d55b398e907387884a2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:13:39.575979 29879 raft_consensus.cc:1261] T 95e4d2bfb49346d091f236eb8b565193 P d090c183506248a3884279bf1c50dc6a [term 1 FOLLOWER]: Refusing update from remote peer 5e67688cfdb54d55b398e907387884a2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:13:39.576982 30087 consensus_queue.cc:1035] T 95e4d2bfb49346d091f236eb8b565193 P 5e67688cfdb54d55b398e907387884a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1dfcec90ece48028dc7121315b1492a" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 41157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:13:39.577363 30084 consensus_queue.cc:1035] T 95e4d2bfb49346d091f236eb8b565193 P 5e67688cfdb54d55b398e907387884a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d090c183506248a3884279bf1c50dc6a" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 34895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:13:39.605620 29601 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59344:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 15:13:39.605916 29601 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59344:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 15:13:39.609592 29601 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 79ccf4a6ac4343b29a95c22795f142b5: Sending DeleteTablet for 3 replicas of tablet 95e4d2bfb49346d091f236eb8b565193
I20260926 15:13:39.610607 29993 tablet_service.cc:1514] Processing DeleteTablet for tablet 95e4d2bfb49346d091f236eb8b565193 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:13:39 UTC) from {username='kudu', principal='kudu/127.12.34.254@KRBTEST.COM'} at 127.0.0.1:35890
I20260926 15:13:39.610648 29725 tablet_service.cc:1514] Processing DeleteTablet for tablet 95e4d2bfb49346d091f236eb8b565193 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:13:39 UTC) from {username='kudu', principal='kudu/127.12.34.254@KRBTEST.COM'} at 127.0.0.1:40552
I20260926 15:13:39.611641 29859 tablet_service.cc:1514] Processing DeleteTablet for tablet 95e4d2bfb49346d091f236eb8b565193 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:13:39 UTC) from {username='kudu', principal='kudu/127.12.34.254@KRBTEST.COM'} at 127.0.0.1:56076
I20260926 15:13:39.612587 30101 tablet_replica.cc:330] T 95e4d2bfb49346d091f236eb8b565193 P d090c183506248a3884279bf1c50dc6a: stopping tablet replica
I20260926 15:13:39.612814 12427 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 29662
I20260926 15:13:39.613125 30101 raft_consensus.cc:2229] T 95e4d2bfb49346d091f236eb8b565193 P d090c183506248a3884279bf1c50dc6a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 15:13:39.613663 30101 raft_consensus.cc:2258] T 95e4d2bfb49346d091f236eb8b565193 P d090c183506248a3884279bf1c50dc6a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:13:39.616432 30101 ts_tablet_manager.cc:1905] T 95e4d2bfb49346d091f236eb8b565193 P d090c183506248a3884279bf1c50dc6a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:13:39.622848 30099 tablet_replica.cc:330] T 95e4d2bfb49346d091f236eb8b565193 P c1dfcec90ece48028dc7121315b1492a: stopping tablet replica
I20260926 15:13:39.623420 30100 tablet_replica.cc:330] T 95e4d2bfb49346d091f236eb8b565193 P 5e67688cfdb54d55b398e907387884a2: stopping tablet replica
I20260926 15:13:39.623960 30099 raft_consensus.cc:2229] T 95e4d2bfb49346d091f236eb8b565193 P c1dfcec90ece48028dc7121315b1492a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 15:13:39.624416 30100 raft_consensus.cc:2229] T 95e4d2bfb49346d091f236eb8b565193 P 5e67688cfdb54d55b398e907387884a2 [term 1 LEADER]: Raft consensus shutting down.
I20260926 15:13:39.624984 30101 ts_tablet_manager.cc:1918] T 95e4d2bfb49346d091f236eb8b565193 P d090c183506248a3884279bf1c50dc6a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 15:13:39.625227 30101 log.cc:1196] T 95e4d2bfb49346d091f236eb8b565193 P d090c183506248a3884279bf1c50dc6a: Deleting WAL directory at /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435318278681-12427-0/minicluster-data/ts-1/wal/wals/95e4d2bfb49346d091f236eb8b565193
I20260926 15:13:39.625480 30099 raft_consensus.cc:2258] T 95e4d2bfb49346d091f236eb8b565193 P c1dfcec90ece48028dc7121315b1492a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:13:39.625859 30101 ts_tablet_manager.cc:1939] T 95e4d2bfb49346d091f236eb8b565193 P d090c183506248a3884279bf1c50dc6a: Deleting consensus metadata
I20260926 15:13:39.625892 30100 raft_consensus.cc:2258] T 95e4d2bfb49346d091f236eb8b565193 P 5e67688cfdb54d55b398e907387884a2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:13:39.627832 29588 catalog_manager.cc:4841] TS d090c183506248a3884279bf1c50dc6a (127.12.34.194:34895): tablet 95e4d2bfb49346d091f236eb8b565193 (table test-table [id=0ccfae505cc8409bb82c6d074af259f5]) successfully deleted
I20260926 15:13:39.628096 30099 ts_tablet_manager.cc:1905] T 95e4d2bfb49346d091f236eb8b565193 P c1dfcec90ece48028dc7121315b1492a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:13:39.628183 30100 ts_tablet_manager.cc:1905] T 95e4d2bfb49346d091f236eb8b565193 P 5e67688cfdb54d55b398e907387884a2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:13:39.632568 30099 ts_tablet_manager.cc:1918] T 95e4d2bfb49346d091f236eb8b565193 P c1dfcec90ece48028dc7121315b1492a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 15:13:39.632731 30099 log.cc:1196] T 95e4d2bfb49346d091f236eb8b565193 P c1dfcec90ece48028dc7121315b1492a: Deleting WAL directory at /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435318278681-12427-0/minicluster-data/ts-2/wal/wals/95e4d2bfb49346d091f236eb8b565193
I20260926 15:13:39.633430 30099 ts_tablet_manager.cc:1939] T 95e4d2bfb49346d091f236eb8b565193 P c1dfcec90ece48028dc7121315b1492a: Deleting consensus metadata
I20260926 15:13:39.635485 29586 catalog_manager.cc:4841] TS c1dfcec90ece48028dc7121315b1492a (127.12.34.195:41157): tablet 95e4d2bfb49346d091f236eb8b565193 (table test-table [id=0ccfae505cc8409bb82c6d074af259f5]) successfully deleted
I20260926 15:13:39.637012 30100 ts_tablet_manager.cc:1918] T 95e4d2bfb49346d091f236eb8b565193 P 5e67688cfdb54d55b398e907387884a2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 15:13:39.637319 30100 log.cc:1196] T 95e4d2bfb49346d091f236eb8b565193 P 5e67688cfdb54d55b398e907387884a2: Deleting WAL directory at /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790435318278681-12427-0/minicluster-data/ts-0/wal/wals/95e4d2bfb49346d091f236eb8b565193
I20260926 15:13:39.637941 30100 ts_tablet_manager.cc:1939] T 95e4d2bfb49346d091f236eb8b565193 P 5e67688cfdb54d55b398e907387884a2: Deleting consensus metadata
I20260926 15:13:39.639631 29587 catalog_manager.cc:4841] TS 5e67688cfdb54d55b398e907387884a2 (127.12.34.193:37247): tablet 95e4d2bfb49346d091f236eb8b565193 (table test-table [id=0ccfae505cc8409bb82c6d074af259f5]) successfully deleted
Sep 26 15:13:39 dist-test-slave-865v krb5kdc[29556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435617, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.34.193@KRBTEST.COM
I20260926 15:13:39.650770 30073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:39.616807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40566 (local address 127.12.34.193:37247)
0926 15:13:39.616977 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0926 15:13:39.616985 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 15:13:39.635316 (+ 18331us) server_negotiation.cc:407] Connection header received
0926 15:13:39.635554 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:13:39.635560 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:13:39.635644 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:13:39.635745 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:13:39.637049 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:39.638595 (+ 1546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:39.640128 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:39.640592 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:39.640724 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:13:39.644964 (+ 4240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:13:39.644975 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:13:39.644989 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:13:39.645019 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:13:39.648013 (+ 2994us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:13:39.648667 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:13:39.648679 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:13:39.648682 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:13:39.648831 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:13:39.649246 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:13:39.649252 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:13:39.649255 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:13:39.649592 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:13:39.649893 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0926 15:13:39.650209 (+ 316us) server_negotiation.cc:299] Negotiation successful
0926 15:13:39.650451 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 15:13:39.651387 29785 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:13:39.755766 12427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8uoJGv/build/asan/bin/kudu with pid 29662
I20260926 15:13:39.774575 12427 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 29796
Sep 26 15:13:39 dist-test-slave-865v krb5kdc[29556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435617, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.34.194@KRBTEST.COM
I20260926 15:13:39.790052 30074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:39.775361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56084 (local address 127.12.34.194:34895)
0926 15:13:39.775576 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0926 15:13:39.775582 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:13:39.775601 (+ 19us) server_negotiation.cc:407] Connection header received
0926 15:13:39.775776 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:13:39.775781 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:13:39.775867 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:13:39.775971 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:13:39.777362 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:39.778195 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:39.779474 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:39.779865 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:39.779992 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:13:39.784402 (+ 4410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:13:39.784414 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:13:39.784431 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:13:39.784465 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:13:39.787307 (+ 2842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:13:39.787967 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:13:39.787975 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:13:39.787977 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:13:39.788110 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:13:39.788443 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:13:39.788446 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:13:39.788448 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:13:39.788818 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:13:39.789045 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0926 15:13:39.789478 (+ 433us) server_negotiation.cc:299] Negotiation successful
0926 15:13:39.789660 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 15:13:39.790707 29919 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:13:39.880857 12427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8uoJGv/build/asan/bin/kudu with pid 29796
I20260926 15:13:39.898121 12427 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 29930
Sep 26 15:13:39 dist-test-slave-865v krb5kdc[29556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435617, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.34.195@KRBTEST.COM
I20260926 15:13:39.914552 30072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:39.898819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35898 (local address 127.12.34.195:41157)
0926 15:13:39.899079 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0926 15:13:39.899084 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:13:39.899102 (+ 18us) server_negotiation.cc:407] Connection header received
0926 15:13:39.899158 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:13:39.899162 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:13:39.899233 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:13:39.899332 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:13:39.901158 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:39.901934 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:39.903345 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:39.903754 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:39.903898 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:13:39.909022 (+ 5124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:13:39.909047 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:13:39.909063 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:13:39.909161 (+ 98us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:13:39.911974 (+ 2813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:13:39.912689 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:13:39.912696 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:13:39.912698 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:13:39.912858 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:13:39.913325 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:13:39.913329 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:13:39.913331 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:13:39.913678 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:13:39.913908 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0926 15:13:39.914131 (+ 223us) server_negotiation.cc:299] Negotiation successful
0926 15:13:39.914313 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135}
I20260926 15:13:39.915195 30053 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:13:40.004163 12427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8uoJGv/build/asan/bin/kudu with pid 29930
I20260926 15:13:40.020627 12427 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 29569
I20260926 15:13:40.021731 29631 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:13:40.138129 12427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8uoJGv/build/asan/bin/kudu with pid 29569
2026-09-26T15:13:40Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4190 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-26T15:13:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T15:13:40Z Disabled control of system clock
I20260926 15:13:40.220216 12427 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8uoJGv/build/asan/bin/kudu
/tmp/dist-test-task8uoJGv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790435318278681-12427-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790435318278681-12427-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790435318278681-12427-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.34.254:38717
--webserver_interface=127.12.34.254
--webserver_port=0
--builtin_ntp_servers=127.12.34.212:43413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.34.254:38717
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 15:13:40.550592 30110 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:13:40.550904 30110 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:13:40.550987 30110 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:13:40.558436 30110 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:13:40.558549 30110 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:13:40.558588 30110 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:13:40.558638 30110 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:13:40.558682 30110 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:13:40.570155 30110 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.34.212:43413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790435318278681-12427-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790435318278681-12427-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.34.254:38717
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.34.254:38717
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.34.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790435318278681-12427-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:13:40.571894 30110 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:13:40.574441 30110 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:13:40.589624 30118 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:13:40.589715 30116 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:13:40.590446 30115 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:13:40.591080 30110 server_base.cc:1034] running on GCE node
I20260926 15:13:40.591952 30110 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:13:40.593895 30110 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:13:40.595160 30110 hybrid_clock.cc:648] HybridClock initialized: now 1790435620595062 us; error 49 us; skew 500 ppm
I20260926 15:13:40.595616 30110 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 15:13:40.598443 30110 webserver.cc:460] Webserver started at http://127.12.34.254:41111/ using document root <none> and password file <none>
I20260926 15:13:40.599159 30110 fs_manager.cc:359] Metadata directory not provided
I20260926 15:13:40.599287 30110 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:13:40.599632 30110 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:13:40.603315 30110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790435318278681-12427-0/minicluster-data/master-0/data/instance:
uuid: "a3e2ff920f9a467591f4a87a4946a124"
format_stamp: "Formatted at 2026-09-26 15:13:40 on dist-test-slave-865v"
server_key: "a6360f6ae788b6f019aba24004e5176f"
server_key_iv: "0e3a78fd74c017391abc59d70f608749"
server_key_version: "encryptionkey@0"
I20260926 15:13:40.604127 30110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790435318278681-12427-0/minicluster-data/master-0/wal/instance:
uuid: "a3e2ff920f9a467591f4a87a4946a124"
format_stamp: "Formatted at 2026-09-26 15:13:40 on dist-test-slave-865v"
server_key: "a6360f6ae788b6f019aba24004e5176f"
server_key_iv: "0e3a78fd74c017391abc59d70f608749"
server_key_version: "encryptionkey@0"
I20260926 15:13:40.609470 30110 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.003s
I20260926 15:13:40.612840 30124 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:13:40.614248 30110 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 15:13:40.614452 30110 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790435318278681-12427-0/minicluster-data/master-0/data,/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790435318278681-12427-0/minicluster-data/master-0/wal
uuid: "a3e2ff920f9a467591f4a87a4946a124"
format_stamp: "Formatted at 2026-09-26 15:13:40 on dist-test-slave-865v"
server_key: "a6360f6ae788b6f019aba24004e5176f"
server_key_iv: "0e3a78fd74c017391abc59d70f608749"
server_key_version: "encryptionkey@0"
I20260926 15:13:40.614727 30110 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790435318278681-12427-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790435318278681-12427-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790435318278681-12427-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:13:40.631170 30110 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:13:40.631943 30110 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:13:40.632215 30110 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:13:40.650614 30110 rpc_server.cc:307] RPC server started. Bound to: 127.12.34.254:38717
I20260926 15:13:40.650635 30175 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.34.254:38717 every 8 connection(s)
I20260926 15:13:40.651866 30110 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
I20260926 15:13:40.655465 30176 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:13:40.657027 12427 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8uoJGv/build/asan/bin/kudu as pid 30110
I20260926 15:13:40.657217 12427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790435318278681-12427-0/minicluster-data/master-0/wal/instance
I20260926 15:13:40.657655 12427 external_mini_cluster.cc:1428] Setting key 8c1c2540cda29cda3381886a2ecf3d45
I20260926 15:13:40.666904 30176 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3e2ff920f9a467591f4a87a4946a124: Bootstrap starting.
I20260926 15:13:40.670454 30176 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a3e2ff920f9a467591f4a87a4946a124: Neither blocks nor log segments found. Creating new log.
I20260926 15:13:40.671277 30179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:40.659494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50200 (local address 127.12.34.254:38717)
0926 15:13:40.660474 (+ 980us) server_negotiation.cc:206] Beginning negotiation
0926 15:13:40.660484 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 15:13:40.660525 (+ 41us) server_negotiation.cc:407] Connection header received
0926 15:13:40.662171 (+ 1646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:13:40.662192 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:13:40.662590 (+ 398us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:13:40.662887 (+ 297us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:13:40.663523 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:40.664910 (+ 1387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:40.666817 (+ 1907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:40.667562 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:40.667703 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:13:40.668923 (+ 1220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:13:40.668931 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:13:40.668944 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:13:40.669522 (+ 578us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:13:40.669614 (+ 92us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:13:40.669634 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:13:40.669726 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 15:13:40.669969 (+ 243us) server_negotiation.cc:299] Negotiation successful
0926 15:13:40.670319 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":596,"thread_start_us":389,"threads_started":1}
I20260926 15:13:40.671504 30176 log.cc:824] T 00000000000000000000000000000000 P a3e2ff920f9a467591f4a87a4946a124: Log is configured to *not* fsync() on all Append() calls
I20260926 15:13:40.674808 30176 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3e2ff920f9a467591f4a87a4946a124: No bootstrap required, opened a new log
I20260926 15:13:40.682637 30176 raft_consensus.cc:348] T 00000000000000000000000000000000 P a3e2ff920f9a467591f4a87a4946a124 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3e2ff920f9a467591f4a87a4946a124" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 38717 } }
I20260926 15:13:40.683019 30176 raft_consensus.cc:374] T 00000000000000000000000000000000 P a3e2ff920f9a467591f4a87a4946a124 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:13:40.683137 30176 raft_consensus.cc:729] T 00000000000000000000000000000000 P a3e2ff920f9a467591f4a87a4946a124 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3e2ff920f9a467591f4a87a4946a124, State: Initialized, Role: FOLLOWER
I20260926 15:13:40.683737 30176 consensus_queue.cc:260] T 00000000000000000000000000000000 P a3e2ff920f9a467591f4a87a4946a124 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3e2ff920f9a467591f4a87a4946a124" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 38717 } }
I20260926 15:13:40.683983 30176 raft_consensus.cc:388] T 00000000000000000000000000000000 P a3e2ff920f9a467591f4a87a4946a124 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 15:13:40.684132 30176 raft_consensus.cc:482] T 00000000000000000000000000000000 P a3e2ff920f9a467591f4a87a4946a124 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 15:13:40.684317 30176 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a3e2ff920f9a467591f4a87a4946a124 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:13:40.686484 30176 raft_consensus.cc:504] T 00000000000000000000000000000000 P a3e2ff920f9a467591f4a87a4946a124 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3e2ff920f9a467591f4a87a4946a124" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 38717 } }
I20260926 15:13:40.686991 30176 leader_election.cc:302] T 00000000000000000000000000000000 P a3e2ff920f9a467591f4a87a4946a124 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a3e2ff920f9a467591f4a87a4946a124; no voters:
I20260926 15:13:40.687575 30176 leader_election.cc:288] T 00000000000000000000000000000000 P a3e2ff920f9a467591f4a87a4946a124 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 15:13:40.687737 30181 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a3e2ff920f9a467591f4a87a4946a124 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:13:40.688941 30181 raft_consensus.cc:686] T 00000000000000000000000000000000 P a3e2ff920f9a467591f4a87a4946a124 [term 1 LEADER]: Becoming Leader. State: Replica: a3e2ff920f9a467591f4a87a4946a124, State: Running, Role: LEADER
I20260926 15:13:40.689491 30181 consensus_queue.cc:237] T 00000000000000000000000000000000 P a3e2ff920f9a467591f4a87a4946a124 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3e2ff920f9a467591f4a87a4946a124" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 38717 } }
I20260926 15:13:40.689813 30176 sys_catalog.cc:564] T 00000000000000000000000000000000 P a3e2ff920f9a467591f4a87a4946a124 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:13:40.691898 30182 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3e2ff920f9a467591f4a87a4946a124 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a3e2ff920f9a467591f4a87a4946a124" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3e2ff920f9a467591f4a87a4946a124" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 38717 } } }
I20260926 15:13:40.692222 30182 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3e2ff920f9a467591f4a87a4946a124 [sys.catalog]: This master's current role is: LEADER
I20260926 15:13:40.692776 30183 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3e2ff920f9a467591f4a87a4946a124 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a3e2ff920f9a467591f4a87a4946a124. Latest consensus state: current_term: 1 leader_uuid: "a3e2ff920f9a467591f4a87a4946a124" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3e2ff920f9a467591f4a87a4946a124" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 38717 } } }
I20260926 15:13:40.693616 30183 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3e2ff920f9a467591f4a87a4946a124 [sys.catalog]: This master's current role is: LEADER
I20260926 15:13:40.695487 30187 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:13:40.700031 30187 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 15:13:40.709276 30187 catalog_manager.cc:1347] Generated new cluster ID: 83f96343aa8a4a798c5c5efcc627f3d8
I20260926 15:13:40.709383 30187 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 15:13:40.727843 30187 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 15:13:40.729296 30187 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:13:40.734766 30187 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a3e2ff920f9a467591f4a87a4946a124: Generated new TSK 0
I20260926 15:13:40.735574 30187 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 15:13:40.740228 12427 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8uoJGv/build/asan/bin/kudu
/tmp/dist-test-task8uoJGv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790435318278681-12427-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790435318278681-12427-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790435318278681-12427-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790435318278681-12427-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.34.193:0
--local_ip_for_outbound_sockets=127.12.34.193
--webserver_interface=127.12.34.193
--webserver_port=0
--tserver_master_addrs=127.12.34.254:38717
--builtin_ntp_servers=127.12.34.212:43413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 15:13:41.075388 30201 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:13:41.075640 30201 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:13:41.075773 30201 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:13:41.082952 30201 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:13:41.083046 30201 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:13:41.083210 30201 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.34.193
I20260926 15:13:41.092288 30201 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.34.212:43413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790435318278681-12427-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790435318278681-12427-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.34.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790435318278681-12427-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.34.193
--webserver_port=0
--tserver_master_addrs=127.12.34.254:38717
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.34.193
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790435318278681-12427-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:13:41.093636 30201 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:13:41.095285 30201 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:13:41.108296 30206 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:13:41.108315 30207 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:13:41.108490 30209 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:13:41.109040 30201 server_base.cc:1034] running on GCE node
I20260926 15:13:41.109598 30201 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:13:41.110412 30201 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:13:41.111608 30201 hybrid_clock.cc:648] HybridClock initialized: now 1790435621111570 us; error 45 us; skew 500 ppm
I20260926 15:13:41.111917 30201 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 15:13:41.114087 30201 webserver.cc:460] Webserver started at http://127.12.34.193:34999/ using document root <none> and password file <none>
I20260926 15:13:41.114727 30201 fs_manager.cc:359] Metadata directory not provided
I20260926 15:13:41.114842 30201 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:13:41.115226 30201 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:13:41.118831 30201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790435318278681-12427-0/minicluster-data/ts-0/data/instance:
uuid: "4e149bbb38694cecb33d8bf6b6f729af"
format_stamp: "Formatted at 2026-09-26 15:13:41 on dist-test-slave-865v"
server_key: "45fd2ab156350069d115867f78750ce9"
server_key_iv: "28d602a81e56b4641d2f3dbd307f479a"
server_key_version: "encryptionkey@0"
I20260926 15:13:41.119738 30201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790435318278681-12427-0/minicluster-data/ts-0/wal/instance:
uuid: "4e149bbb38694cecb33d8bf6b6f729af"
format_stamp: "Formatted at 2026-09-26 15:13:41 on dist-test-slave-865v"
server_key: "45fd2ab156350069d115867f78750ce9"
server_key_iv: "28d602a81e56b4641d2f3dbd307f479a"
server_key_version: "encryptionkey@0"
I20260926 15:13:41.126425 30201 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260926 15:13:41.129958 30215 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:13:41.131366 30201 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 15:13:41.131532 30201 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790435318278681-12427-0/minicluster-data/ts-0/data,/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790435318278681-12427-0/minicluster-data/ts-0/wal
uuid: "4e149bbb38694cecb33d8bf6b6f729af"
format_stamp: "Formatted at 2026-09-26 15:13:41 on dist-test-slave-865v"
server_key: "45fd2ab156350069d115867f78750ce9"
server_key_iv: "28d602a81e56b4641d2f3dbd307f479a"
server_key_version: "encryptionkey@0"
I20260926 15:13:41.131742 30201 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790435318278681-12427-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790435318278681-12427-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790435318278681-12427-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:13:41.148242 30201 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:13:41.149008 30201 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:13:41.149343 30201 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:13:41.150269 30201 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:13:41.151659 30201 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:13:41.151770 30201 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:13:41.151902 30201 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:13:41.152001 30201 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:13:41.184334 30201 rpc_server.cc:307] RPC server started. Bound to: 127.12.34.193:33681
I20260926 15:13:41.184427 30327 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.34.193:33681 every 8 connection(s)
I20260926 15:13:41.185691 30201 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790435318278681-12427-0/minicluster-data/ts-0/data/info.pb
I20260926 15:13:41.187230 12427 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8uoJGv/build/asan/bin/kudu as pid 30201
I20260926 15:13:41.187403 12427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790435318278681-12427-0/minicluster-data/ts-0/wal/instance
I20260926 15:13:41.187862 12427 external_mini_cluster.cc:1428] Setting key 6fd7009b7c1f2a43fb3fac55525f26c3
I20260926 15:13:41.190825 12427 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8uoJGv/build/asan/bin/kudu
/tmp/dist-test-task8uoJGv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790435318278681-12427-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790435318278681-12427-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790435318278681-12427-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790435318278681-12427-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.34.194:0
--local_ip_for_outbound_sockets=127.12.34.194
--webserver_interface=127.12.34.194
--webserver_port=0
--tserver_master_addrs=127.12.34.254:38717
--builtin_ntp_servers=127.12.34.212:43413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 15:13:41.198957 30330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:41.188442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.193:46169 (local address 127.12.34.254:38717)
0926 15:13:41.188928 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0926 15:13:41.188935 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:13:41.190250 (+ 1315us) server_negotiation.cc:407] Connection header received
0926 15:13:41.192110 (+ 1860us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:13:41.192115 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:13:41.192198 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:13:41.192329 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:13:41.193385 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:41.194243 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:41.196037 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:41.196456 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:41.196585 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:13:41.198007 (+ 1422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:13:41.198016 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:13:41.198021 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:13:41.198046 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:13:41.198106 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:13:41.198124 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:13:41.198228 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 15:13:41.198446 (+ 218us) server_negotiation.cc:299] Negotiation successful
0926 15:13:41.198550 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":263,"threads_started":1}
I20260926 15:13:41.199787 30331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:41.189068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:38717 (local address 127.12.34.193:46169)
0926 15:13:41.190110 (+ 1042us) negotiation.cc:107] Waiting for socket to connect
0926 15:13:41.190144 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 15:13:41.191869 (+ 1725us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:13:41.192546 (+ 677us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:13:41.192554 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:13:41.192634 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:13:41.193223 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:13:41.193240 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:13:41.194403 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:13:41.194408 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:13:41.195895 (+ 1487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:13:41.195906 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:13:41.196682 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:13:41.196690 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:13:41.196919 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:13:41.197703 (+ 784us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 15:13:41.197706 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:13:41.197747 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 15:13:41.197820 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:13:41.198241 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:13:41.198249 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:13:41.198255 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 15:13:41.198389 (+ 134us) client_negotiation.cc:240] Negotiation successful
0926 15:13:41.198655 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":292,"threads_started":1}
I20260926 15:13:41.201375 30328 heartbeater.cc:347] Connected to a master server at 127.12.34.254:38717
I20260926 15:13:41.201736 30328 heartbeater.cc:464] Registering TS with master...
I20260926 15:13:41.202490 30328 heartbeater.cc:511] Master 127.12.34.254:38717 requested a full tablet report, sending...
I20260926 15:13:41.204408 30141 ts_manager.cc:194] Registered new tserver with Master: 4e149bbb38694cecb33d8bf6b6f729af (127.12.34.193:33681)
I20260926 15:13:41.205906 30141 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.34.193:46169
W20260926 15:13:41.511406 30332 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:13:41.511672 30332 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:13:41.511829 30332 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:13:41.519117 30332 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:13:41.519207 30332 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:13:41.519403 30332 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.34.194
I20260926 15:13:41.528220 30332 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.34.212:43413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790435318278681-12427-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790435318278681-12427-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.34.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790435318278681-12427-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.34.194
--webserver_port=0
--tserver_master_addrs=127.12.34.254:38717
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.34.194
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790435318278681-12427-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:13:41.529420 30332 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:13:41.531111 30332 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:13:41.543998 30337 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:13:41.543967 30338 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:13:41.543938 30340 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:13:41.544704 30332 server_base.cc:1034] running on GCE node
I20260926 15:13:41.545392 30332 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:13:41.546262 30332 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:13:41.547493 30332 hybrid_clock.cc:648] HybridClock initialized: now 1790435621547455 us; error 44 us; skew 500 ppm
I20260926 15:13:41.547803 30332 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 15:13:41.550227 30332 webserver.cc:460] Webserver started at http://127.12.34.194:46671/ using document root <none> and password file <none>
I20260926 15:13:41.550884 30332 fs_manager.cc:359] Metadata directory not provided
I20260926 15:13:41.550966 30332 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:13:41.551293 30332 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:13:41.554026 30332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790435318278681-12427-0/minicluster-data/ts-1/data/instance:
uuid: "a4b982ef2da84c9082468c2f052957c6"
format_stamp: "Formatted at 2026-09-26 15:13:41 on dist-test-slave-865v"
server_key: "b747d4450a2b8062ae987a69defd490b"
server_key_iv: "61a73d684077edd5e7d4642b8190fdae"
server_key_version: "encryptionkey@0"
I20260926 15:13:41.554797 30332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790435318278681-12427-0/minicluster-data/ts-1/wal/instance:
uuid: "a4b982ef2da84c9082468c2f052957c6"
format_stamp: "Formatted at 2026-09-26 15:13:41 on dist-test-slave-865v"
server_key: "b747d4450a2b8062ae987a69defd490b"
server_key_iv: "61a73d684077edd5e7d4642b8190fdae"
server_key_version: "encryptionkey@0"
I20260926 15:13:41.559520 30332 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260926 15:13:41.562832 30346 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:13:41.564059 30332 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 15:13:41.564227 30332 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790435318278681-12427-0/minicluster-data/ts-1/data,/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790435318278681-12427-0/minicluster-data/ts-1/wal
uuid: "a4b982ef2da84c9082468c2f052957c6"
format_stamp: "Formatted at 2026-09-26 15:13:41 on dist-test-slave-865v"
server_key: "b747d4450a2b8062ae987a69defd490b"
server_key_iv: "61a73d684077edd5e7d4642b8190fdae"
server_key_version: "encryptionkey@0"
I20260926 15:13:41.564431 30332 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790435318278681-12427-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790435318278681-12427-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790435318278681-12427-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:13:41.579473 30332 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:13:41.580287 30332 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:13:41.580595 30332 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:13:41.581596 30332 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:13:41.582993 30332 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:13:41.583115 30332 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:13:41.583247 30332 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:13:41.583333 30332 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:13:41.616681 30332 rpc_server.cc:307] RPC server started. Bound to: 127.12.34.194:45131
I20260926 15:13:41.616737 30458 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.34.194:45131 every 8 connection(s)
I20260926 15:13:41.617996 30332 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790435318278681-12427-0/minicluster-data/ts-1/data/info.pb
I20260926 15:13:41.625124 12427 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8uoJGv/build/asan/bin/kudu as pid 30332
I20260926 15:13:41.625308 12427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790435318278681-12427-0/minicluster-data/ts-1/wal/instance
I20260926 15:13:41.625777 12427 external_mini_cluster.cc:1428] Setting key 9d6dfe6f2001aa4884b25043f4d76321
I20260926 15:13:41.628777 12427 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8uoJGv/build/asan/bin/kudu
/tmp/dist-test-task8uoJGv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790435318278681-12427-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790435318278681-12427-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790435318278681-12427-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790435318278681-12427-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.34.195:0
--local_ip_for_outbound_sockets=127.12.34.195
--webserver_interface=127.12.34.195
--webserver_port=0
--tserver_master_addrs=127.12.34.254:38717
--builtin_ntp_servers=127.12.34.212:43413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 15:13:41.630740 30330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:41.620415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.194:59571 (local address 127.12.34.254:38717)
0926 15:13:41.620592 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0926 15:13:41.620596 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:13:41.621904 (+ 1308us) server_negotiation.cc:407] Connection header received
0926 15:13:41.623591 (+ 1687us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:13:41.623596 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:13:41.623664 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:13:41.623761 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:13:41.624751 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:41.625678 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:41.627685 (+ 2007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:41.628299 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:41.628431 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:13:41.629814 (+ 1383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:13:41.629832 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:13:41.629839 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:13:41.629864 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:13:41.629907 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:13:41.629925 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:13:41.630024 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 15:13:41.630219 (+ 195us) server_negotiation.cc:299] Negotiation successful
0926 15:13:41.630327 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 15:13:41.631399 30461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:41.620917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:38717 (local address 127.12.34.194:59571)
0926 15:13:41.621747 (+ 830us) negotiation.cc:107] Waiting for socket to connect
0926 15:13:41.621785 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 15:13:41.623340 (+ 1555us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:13:41.623947 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:13:41.623954 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:13:41.624030 (+ 76us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:13:41.624568 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:13:41.624583 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:13:41.625851 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:13:41.625856 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:13:41.627542 (+ 1686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:13:41.627554 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:13:41.628499 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:13:41.628504 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:13:41.628688 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:13:41.629527 (+ 839us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 15:13:41.629529 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:13:41.629563 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 15:13:41.629635 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:13:41.630027 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:13:41.630034 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:13:41.630039 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 15:13:41.630172 (+ 133us) client_negotiation.cc:240] Negotiation successful
0926 15:13:41.630422 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":271,"threads_started":1}
I20260926 15:13:41.632675 30459 heartbeater.cc:347] Connected to a master server at 127.12.34.254:38717
I20260926 15:13:41.632970 30459 heartbeater.cc:464] Registering TS with master...
I20260926 15:13:41.633792 30459 heartbeater.cc:511] Master 127.12.34.254:38717 requested a full tablet report, sending...
I20260926 15:13:41.635332 30141 ts_manager.cc:194] Registered new tserver with Master: a4b982ef2da84c9082468c2f052957c6 (127.12.34.194:45131)
I20260926 15:13:41.636272 30141 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.34.194:59571
W20260926 15:13:41.953207 30462 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:13:41.953465 30462 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:13:41.953634 30462 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:13:41.960563 30462 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:13:41.960642 30462 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:13:41.960807 30462 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.34.195
I20260926 15:13:41.969384 30462 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.34.212:43413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790435318278681-12427-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790435318278681-12427-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.34.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790435318278681-12427-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.34.195
--webserver_port=0
--tserver_master_addrs=127.12.34.254:38717
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.34.195
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790435318278681-12427-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:13:41.970533 30462 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:13:41.972185 30462 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:13:41.984952 30468 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:13:41.984989 30467 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:13:41.984866 30470 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:13:41.985615 30462 server_base.cc:1034] running on GCE node
I20260926 15:13:41.986234 30462 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:13:41.987066 30462 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:13:41.988270 30462 hybrid_clock.cc:648] HybridClock initialized: now 1790435621988221 us; error 43 us; skew 500 ppm
I20260926 15:13:41.988593 30462 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 15:13:41.990888 30462 webserver.cc:460] Webserver started at http://127.12.34.195:45435/ using document root <none> and password file <none>
I20260926 15:13:41.991537 30462 fs_manager.cc:359] Metadata directory not provided
I20260926 15:13:41.991644 30462 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:13:41.991946 30462 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:13:41.994679 30462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790435318278681-12427-0/minicluster-data/ts-2/data/instance:
uuid: "bff6b8c127564c09853a756f7d126697"
format_stamp: "Formatted at 2026-09-26 15:13:41 on dist-test-slave-865v"
server_key: "52f150e2778955ed244298ce17933f2e"
server_key_iv: "6945680b23f38661912d9c2fa8161eb1"
server_key_version: "encryptionkey@0"
I20260926 15:13:41.995474 30462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790435318278681-12427-0/minicluster-data/ts-2/wal/instance:
uuid: "bff6b8c127564c09853a756f7d126697"
format_stamp: "Formatted at 2026-09-26 15:13:41 on dist-test-slave-865v"
server_key: "52f150e2778955ed244298ce17933f2e"
server_key_iv: "6945680b23f38661912d9c2fa8161eb1"
server_key_version: "encryptionkey@0"
I20260926 15:13:42.000211 30462 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260926 15:13:42.003736 30476 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:13:42.004968 30462 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 15:13:42.005172 30462 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790435318278681-12427-0/minicluster-data/ts-2/data,/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790435318278681-12427-0/minicluster-data/ts-2/wal
uuid: "bff6b8c127564c09853a756f7d126697"
format_stamp: "Formatted at 2026-09-26 15:13:41 on dist-test-slave-865v"
server_key: "52f150e2778955ed244298ce17933f2e"
server_key_iv: "6945680b23f38661912d9c2fa8161eb1"
server_key_version: "encryptionkey@0"
I20260926 15:13:42.005385 30462 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790435318278681-12427-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790435318278681-12427-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790435318278681-12427-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:13:42.022578 30462 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:13:42.023422 30462 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:13:42.023717 30462 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:13:42.024608 30462 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:13:42.026001 30462 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:13:42.026121 30462 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:13:42.026245 30462 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:13:42.026330 30462 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:13:42.060189 30462 rpc_server.cc:307] RPC server started. Bound to: 127.12.34.195:38083
I20260926 15:13:42.060237 30588 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.34.195:38083 every 8 connection(s)
I20260926 15:13:42.061574 30462 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790435318278681-12427-0/minicluster-data/ts-2/data/info.pb
I20260926 15:13:42.065011 12427 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8uoJGv/build/asan/bin/kudu as pid 30462
I20260926 15:13:42.065228 12427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790435318278681-12427-0/minicluster-data/ts-2/wal/instance
I20260926 15:13:42.070665 12427 external_mini_cluster.cc:1428] Setting key 78db7ac85da37fc70e68b2e43db91504
I20260926 15:13:42.077840 30330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:42.064519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.195:44689 (local address 127.12.34.254:38717)
0926 15:13:42.069298 (+ 4779us) server_negotiation.cc:206] Beginning negotiation
0926 15:13:42.069303 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:13:42.069324 (+ 21us) server_negotiation.cc:407] Connection header received
0926 15:13:42.070674 (+ 1350us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:13:42.070679 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:13:42.070792 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:13:42.070889 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:13:42.072299 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:42.073424 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:42.075064 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:42.075460 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:42.075567 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:13:42.076889 (+ 1322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:13:42.076894 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:13:42.076897 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:13:42.076915 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:13:42.076947 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:13:42.076959 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:13:42.077041 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 15:13:42.077416 (+ 375us) server_negotiation.cc:299] Negotiation successful
0926 15:13:42.077500 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4633}
I20260926 15:13:42.078716 30591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:42.065110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:38717 (local address 127.12.34.195:44689)
0926 15:13:42.067322 (+ 2212us) negotiation.cc:107] Waiting for socket to connect
0926 15:13:42.067395 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0926 15:13:42.070348 (+ 2953us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:13:42.071138 (+ 790us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:13:42.071151 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:13:42.071266 (+ 115us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:13:42.072104 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:13:42.072130 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:13:42.073587 (+ 1457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:13:42.073593 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:13:42.074898 (+ 1305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:13:42.074910 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:13:42.075648 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:13:42.075652 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:13:42.075871 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:13:42.076574 (+ 703us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 15:13:42.076577 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:13:42.076616 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 15:13:42.076697 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:13:42.077159 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:13:42.077170 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:13:42.077179 (+ 9us) client_negotiation.cc:769] Sending connection context
0926 15:13:42.077350 (+ 171us) client_negotiation.cc:240] Negotiation successful
0926 15:13:42.077639 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1529,"thread_start_us":1179,"threads_started":1}
I20260926 15:13:42.079990 30589 heartbeater.cc:347] Connected to a master server at 127.12.34.254:38717
I20260926 15:13:42.080292 30589 heartbeater.cc:464] Registering TS with master...
I20260926 15:13:42.081010 30589 heartbeater.cc:511] Master 127.12.34.254:38717 requested a full tablet report, sending...
I20260926 15:13:42.082651 30141 ts_manager.cc:194] Registered new tserver with Master: bff6b8c127564c09853a756f7d126697 (127.12.34.195:38083)
I20260926 15:13:42.083461 30141 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.34.195:44689
I20260926 15:13:42.086104 12427 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 15:13:42.097471 30330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:42.090853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50204 (local address 127.12.34.254:38717)
0926 15:13:42.091095 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0926 15:13:42.091103 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 15:13:42.091658 (+ 555us) server_negotiation.cc:407] Connection header received
0926 15:13:42.091915 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:13:42.091921 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:13:42.092015 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:13:42.092142 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:13:42.092678 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:42.093644 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:42.095102 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:42.095529 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:42.095687 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:13:42.096728 (+ 1041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:13:42.096733 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:13:42.096736 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:13:42.096755 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:13:42.096792 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:13:42.096805 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:13:42.096914 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 15:13:42.097054 (+ 140us) server_negotiation.cc:299] Negotiation successful
0926 15:13:42.097168 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260926 15:13:42.101315 30141 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50204:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 15:13:42.108124 30141 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 15:13:42.142979 30602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:42.119888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.195:38083 (local address 127.0.0.1:56866)
0926 15:13:42.132424 (+ 12536us) negotiation.cc:107] Waiting for socket to connect
0926 15:13:42.132487 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0926 15:13:42.132746 (+ 259us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:13:42.134712 (+ 1966us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:13:42.134718 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:13:42.134748 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:13:42.135170 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:13:42.135182 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:13:42.137260 (+ 2078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:13:42.137265 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:13:42.140332 (+ 3067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:13:42.140352 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:13:42.142018 (+ 1666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:13:42.142025 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:13:42.142228 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:13:42.142285 (+ 57us) client_negotiation.cc:769] Sending connection context
0926 15:13:42.142419 (+ 134us) client_negotiation.cc:240] Negotiation successful
0926 15:13:42.142599 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12116,"thread_start_us":13975,"threads_started":1}
I20260926 15:13:42.143533 30604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:42.120860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50922 (local address 127.12.34.194:45131)
0926 15:13:42.132944 (+ 12084us) server_negotiation.cc:206] Beginning negotiation
0926 15:13:42.132954 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 15:13:42.134256 (+ 1302us) server_negotiation.cc:407] Connection header received
0926 15:13:42.134550 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:13:42.134557 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:13:42.134764 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:13:42.134942 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:13:42.135914 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:42.137371 (+ 1457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:42.140667 (+ 3296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:42.142118 (+ 1451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:42.142219 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:13:42.142457 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0926 15:13:42.143063 (+ 606us) server_negotiation.cc:299] Negotiation successful
0926 15:13:42.143233 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11912,"thread_start_us":12066,"threads_started":1}
I20260926 15:13:42.143548 30603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:42.120201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56866 (local address 127.12.34.195:38083)
0926 15:13:42.133768 (+ 13567us) server_negotiation.cc:206] Beginning negotiation
0926 15:13:42.133778 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 15:13:42.133816 (+ 38us) server_negotiation.cc:407] Connection header received
0926 15:13:42.133923 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:13:42.133932 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:13:42.134206 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:13:42.134406 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:13:42.135393 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:42.137054 (+ 1661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:42.140552 (+ 3498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:42.141821 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:42.141984 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:13:42.142274 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0926 15:13:42.143013 (+ 739us) server_negotiation.cc:299] Negotiation successful
0926 15:13:42.143201 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13387,"thread_start_us":13237,"threads_started":1}
I20260926 15:13:42.143888 30601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:42.120045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.194:45131 (local address 127.0.0.1:50922)
0926 15:13:42.134137 (+ 14092us) negotiation.cc:107] Waiting for socket to connect
0926 15:13:42.134156 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 15:13:42.134369 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:13:42.135191 (+ 822us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:13:42.135194 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:13:42.135224 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:13:42.135689 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:13:42.135701 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:13:42.137570 (+ 1869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:13:42.137577 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:13:42.140437 (+ 2860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:13:42.140451 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:13:42.142629 (+ 2178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:13:42.142635 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:13:42.142839 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:13:42.142906 (+ 67us) client_negotiation.cc:769] Sending connection context
0926 15:13:42.143005 (+ 99us) client_negotiation.cc:240] Negotiation successful
0926 15:13:42.143434 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13964,"mutex_wait_us":4,"thread_start_us":12170,"threads_started":1}
I20260926 15:13:42.147662 30394 tablet_service.cc:1467] Processing CreateTablet for tablet c38d4795bd924553992dbfce24ebc12f (DEFAULT_TABLE table=test-table [id=e7c1bbe79108464aa3adb48abb9e33f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:13:42.147665 30524 tablet_service.cc:1467] Processing CreateTablet for tablet c38d4795bd924553992dbfce24ebc12f (DEFAULT_TABLE table=test-table [id=e7c1bbe79108464aa3adb48abb9e33f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:13:42.149197 30394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c38d4795bd924553992dbfce24ebc12f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:13:42.149197 30524 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c38d4795bd924553992dbfce24ebc12f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:13:42.149467 30605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:42.134330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.193:33681 (local address 127.0.0.1:51036)
0926 15:13:42.140889 (+ 6559us) negotiation.cc:107] Waiting for socket to connect
0926 15:13:42.140910 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 15:13:42.141375 (+ 465us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:13:42.142773 (+ 1398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:13:42.142779 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:13:42.142835 (+ 56us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:13:42.144373 (+ 1538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:13:42.144390 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:13:42.145808 (+ 1418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:13:42.145813 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:13:42.147090 (+ 1277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:13:42.147101 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:13:42.148737 (+ 1636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:13:42.148742 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:13:42.148856 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:13:42.148877 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 15:13:42.148986 (+ 109us) client_negotiation.cc:240] Negotiation successful
0926 15:13:42.149129 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6435,"mutex_wait_us":1198,"thread_start_us":6302,"threads_started":1}
I20260926 15:13:42.149912 30606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:42.134580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51036 (local address 127.12.34.193:33681)
0926 15:13:42.139752 (+ 5172us) server_negotiation.cc:206] Beginning negotiation
0926 15:13:42.139760 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 15:13:42.140996 (+ 1236us) server_negotiation.cc:407] Connection header received
0926 15:13:42.141560 (+ 564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:13:42.141569 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:13:42.141817 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:13:42.142489 (+ 672us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:13:42.144556 (+ 2067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:42.145671 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:42.147270 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:42.148301 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:42.148442 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:13:42.148632 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0926 15:13:42.149032 (+ 400us) server_negotiation.cc:299] Negotiation successful
0926 15:13:42.149221 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5026,"thread_start_us":4926,"threads_started":1}
I20260926 15:13:42.153051 30263 tablet_service.cc:1467] Processing CreateTablet for tablet c38d4795bd924553992dbfce24ebc12f (DEFAULT_TABLE table=test-table [id=e7c1bbe79108464aa3adb48abb9e33f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:13:42.154527 30263 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c38d4795bd924553992dbfce24ebc12f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:13:42.165871 30607 tablet_bootstrap.cc:492] T c38d4795bd924553992dbfce24ebc12f P a4b982ef2da84c9082468c2f052957c6: Bootstrap starting.
I20260926 15:13:42.166342 30608 tablet_bootstrap.cc:492] T c38d4795bd924553992dbfce24ebc12f P bff6b8c127564c09853a756f7d126697: Bootstrap starting.
I20260926 15:13:42.167959 30609 tablet_bootstrap.cc:492] T c38d4795bd924553992dbfce24ebc12f P 4e149bbb38694cecb33d8bf6b6f729af: Bootstrap starting.
I20260926 15:13:42.169977 30607 tablet_bootstrap.cc:654] T c38d4795bd924553992dbfce24ebc12f P a4b982ef2da84c9082468c2f052957c6: Neither blocks nor log segments found. Creating new log.
I20260926 15:13:42.170290 30608 tablet_bootstrap.cc:654] T c38d4795bd924553992dbfce24ebc12f P bff6b8c127564c09853a756f7d126697: Neither blocks nor log segments found. Creating new log.
I20260926 15:13:42.171197 30608 log.cc:824] T c38d4795bd924553992dbfce24ebc12f P bff6b8c127564c09853a756f7d126697: Log is configured to *not* fsync() on all Append() calls
I20260926 15:13:42.171222 30607 log.cc:824] T c38d4795bd924553992dbfce24ebc12f P a4b982ef2da84c9082468c2f052957c6: Log is configured to *not* fsync() on all Append() calls
I20260926 15:13:42.171401 30609 tablet_bootstrap.cc:654] T c38d4795bd924553992dbfce24ebc12f P 4e149bbb38694cecb33d8bf6b6f729af: Neither blocks nor log segments found. Creating new log.
I20260926 15:13:42.172636 30609 log.cc:824] T c38d4795bd924553992dbfce24ebc12f P 4e149bbb38694cecb33d8bf6b6f729af: Log is configured to *not* fsync() on all Append() calls
I20260926 15:13:42.174618 30608 tablet_bootstrap.cc:492] T c38d4795bd924553992dbfce24ebc12f P bff6b8c127564c09853a756f7d126697: No bootstrap required, opened a new log
I20260926 15:13:42.174892 30608 ts_tablet_manager.cc:1397] T c38d4795bd924553992dbfce24ebc12f P bff6b8c127564c09853a756f7d126697: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260926 15:13:42.175207 30609 tablet_bootstrap.cc:492] T c38d4795bd924553992dbfce24ebc12f P 4e149bbb38694cecb33d8bf6b6f729af: No bootstrap required, opened a new log
I20260926 15:13:42.175483 30609 ts_tablet_manager.cc:1397] T c38d4795bd924553992dbfce24ebc12f P 4e149bbb38694cecb33d8bf6b6f729af: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260926 15:13:42.176947 30607 tablet_bootstrap.cc:492] T c38d4795bd924553992dbfce24ebc12f P a4b982ef2da84c9082468c2f052957c6: No bootstrap required, opened a new log
I20260926 15:13:42.177223 30607 ts_tablet_manager.cc:1397] T c38d4795bd924553992dbfce24ebc12f P a4b982ef2da84c9082468c2f052957c6: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20260926 15:13:42.183235 30608 raft_consensus.cc:348] T c38d4795bd924553992dbfce24ebc12f P bff6b8c127564c09853a756f7d126697 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bff6b8c127564c09853a756f7d126697" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 38083 } } peers { permanent_uuid: "a4b982ef2da84c9082468c2f052957c6" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 45131 } } peers { permanent_uuid: "4e149bbb38694cecb33d8bf6b6f729af" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 33681 } }
I20260926 15:13:42.184196 30608 raft_consensus.cc:374] T c38d4795bd924553992dbfce24ebc12f P bff6b8c127564c09853a756f7d126697 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:13:42.184350 30608 raft_consensus.cc:729] T c38d4795bd924553992dbfce24ebc12f P bff6b8c127564c09853a756f7d126697 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bff6b8c127564c09853a756f7d126697, State: Initialized, Role: FOLLOWER
I20260926 15:13:42.185204 30608 consensus_queue.cc:260] T c38d4795bd924553992dbfce24ebc12f P bff6b8c127564c09853a756f7d126697 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bff6b8c127564c09853a756f7d126697" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 38083 } } peers { permanent_uuid: "a4b982ef2da84c9082468c2f052957c6" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 45131 } } peers { permanent_uuid: "4e149bbb38694cecb33d8bf6b6f729af" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 33681 } }
I20260926 15:13:42.186506 30609 raft_consensus.cc:348] T c38d4795bd924553992dbfce24ebc12f P 4e149bbb38694cecb33d8bf6b6f729af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bff6b8c127564c09853a756f7d126697" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 38083 } } peers { permanent_uuid: "a4b982ef2da84c9082468c2f052957c6" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 45131 } } peers { permanent_uuid: "4e149bbb38694cecb33d8bf6b6f729af" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 33681 } }
I20260926 15:13:42.186980 30609 raft_consensus.cc:374] T c38d4795bd924553992dbfce24ebc12f P 4e149bbb38694cecb33d8bf6b6f729af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:13:42.187084 30609 raft_consensus.cc:729] T c38d4795bd924553992dbfce24ebc12f P 4e149bbb38694cecb33d8bf6b6f729af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e149bbb38694cecb33d8bf6b6f729af, State: Initialized, Role: FOLLOWER
I20260926 15:13:42.187621 30607 raft_consensus.cc:348] T c38d4795bd924553992dbfce24ebc12f P a4b982ef2da84c9082468c2f052957c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bff6b8c127564c09853a756f7d126697" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 38083 } } peers { permanent_uuid: "a4b982ef2da84c9082468c2f052957c6" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 45131 } } peers { permanent_uuid: "4e149bbb38694cecb33d8bf6b6f729af" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 33681 } }
I20260926 15:13:42.188051 30607 raft_consensus.cc:374] T c38d4795bd924553992dbfce24ebc12f P a4b982ef2da84c9082468c2f052957c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:13:42.188149 30607 raft_consensus.cc:729] T c38d4795bd924553992dbfce24ebc12f P a4b982ef2da84c9082468c2f052957c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4b982ef2da84c9082468c2f052957c6, State: Initialized, Role: FOLLOWER
I20260926 15:13:42.187945 30609 consensus_queue.cc:260] T c38d4795bd924553992dbfce24ebc12f P 4e149bbb38694cecb33d8bf6b6f729af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bff6b8c127564c09853a756f7d126697" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 38083 } } peers { permanent_uuid: "a4b982ef2da84c9082468c2f052957c6" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 45131 } } peers { permanent_uuid: "4e149bbb38694cecb33d8bf6b6f729af" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 33681 } }
I20260926 15:13:42.188930 30607 consensus_queue.cc:260] T c38d4795bd924553992dbfce24ebc12f P a4b982ef2da84c9082468c2f052957c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bff6b8c127564c09853a756f7d126697" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 38083 } } peers { permanent_uuid: "a4b982ef2da84c9082468c2f052957c6" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 45131 } } peers { permanent_uuid: "4e149bbb38694cecb33d8bf6b6f729af" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 33681 } }
I20260926 15:13:42.195343 30607 ts_tablet_manager.cc:1428] T c38d4795bd924553992dbfce24ebc12f P a4b982ef2da84c9082468c2f052957c6: Time spent starting tablet: real 0.018s user 0.015s sys 0.000s
I20260926 15:13:42.196724 30589 heartbeater.cc:503] Master 127.12.34.254:38717 was elected leader, sending a full tablet report...
I20260926 15:13:42.197259 30608 ts_tablet_manager.cc:1428] T c38d4795bd924553992dbfce24ebc12f P bff6b8c127564c09853a756f7d126697: Time spent starting tablet: real 0.022s user 0.016s sys 0.004s
I20260926 15:13:42.197752 30459 heartbeater.cc:503] Master 127.12.34.254:38717 was elected leader, sending a full tablet report...
I20260926 15:13:42.202620 30328 heartbeater.cc:503] Master 127.12.34.254:38717 was elected leader, sending a full tablet report...
I20260926 15:13:42.203836 30609 ts_tablet_manager.cc:1428] T c38d4795bd924553992dbfce24ebc12f P 4e149bbb38694cecb33d8bf6b6f729af: Time spent starting tablet: real 0.028s user 0.016s sys 0.012s
W20260926 15:13:42.313452 30590 tablet.cc:2369] T c38d4795bd924553992dbfce24ebc12f P bff6b8c127564c09853a756f7d126697: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 15:13:42.370112 30460 tablet.cc:2369] T c38d4795bd924553992dbfce24ebc12f P a4b982ef2da84c9082468c2f052957c6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 15:13:42.425735 30615 raft_consensus.cc:482] T c38d4795bd924553992dbfce24ebc12f P a4b982ef2da84c9082468c2f052957c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 15:13:42.426106 30615 raft_consensus.cc:504] T c38d4795bd924553992dbfce24ebc12f P a4b982ef2da84c9082468c2f052957c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bff6b8c127564c09853a756f7d126697" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 38083 } } peers { permanent_uuid: "a4b982ef2da84c9082468c2f052957c6" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 45131 } } peers { permanent_uuid: "4e149bbb38694cecb33d8bf6b6f729af" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 33681 } }
I20260926 15:13:42.427847 30615 leader_election.cc:288] T c38d4795bd924553992dbfce24ebc12f P a4b982ef2da84c9082468c2f052957c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bff6b8c127564c09853a756f7d126697 (127.12.34.195:38083), 4e149bbb38694cecb33d8bf6b6f729af (127.12.34.193:33681)
I20260926 15:13:42.435045 30603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:42.428246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.194:34781 (local address 127.12.34.195:38083)
0926 15:13:42.428418 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0926 15:13:42.428422 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:13:42.429017 (+ 595us) server_negotiation.cc:407] Connection header received
0926 15:13:42.429308 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:13:42.429318 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:13:42.429416 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:13:42.429479 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:13:42.430306 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:42.431181 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:42.433170 (+ 1989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:42.434049 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:42.434173 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:13:42.434249 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 15:13:42.434651 (+ 402us) server_negotiation.cc:299] Negotiation successful
0926 15:13:42.434798 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 15:13:42.435117 30616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:42.428233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.195:38083 (local address 127.12.34.194:34781)
0926 15:13:42.428904 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0926 15:13:42.428925 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 15:13:42.429069 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:13:42.429697 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:13:42.429703 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:13:42.429753 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:13:42.430150 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:13:42.430163 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:13:42.431309 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:13:42.431314 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:13:42.432953 (+ 1639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:13:42.432965 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:13:42.434264 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:13:42.434274 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:13:42.434463 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:13:42.434500 (+ 37us) client_negotiation.cc:769] Sending connection context
0926 15:13:42.434570 (+ 70us) client_negotiation.cc:240] Negotiation successful
0926 15:13:42.434736 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"mutex_wait_us":39,"thread_start_us":359,"threads_started":1}
I20260926 15:13:42.435132 30606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:42.429459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.194:50963 (local address 127.12.34.193:33681)
0926 15:13:42.429660 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0926 15:13:42.429666 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:13:42.429773 (+ 107us) server_negotiation.cc:407] Connection header received
0926 15:13:42.429936 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:13:42.429942 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:13:42.430032 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:13:42.430135 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:13:42.430746 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:42.431533 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:42.433390 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:42.434218 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:42.434363 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:13:42.434431 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 15:13:42.434755 (+ 324us) server_negotiation.cc:299] Negotiation successful
0926 15:13:42.434885 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 15:13:42.435189 30617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:42.429053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.193:33681 (local address 127.12.34.194:50963)
0926 15:13:42.429664 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0926 15:13:42.429678 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 15:13:42.429805 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:13:42.430150 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:13:42.430152 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:13:42.430169 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:13:42.430586 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:13:42.430599 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:13:42.431690 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:13:42.431694 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:13:42.433208 (+ 1514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:13:42.433222 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:13:42.434401 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:13:42.434406 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:13:42.434529 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:13:42.434577 (+ 48us) client_negotiation.cc:769] Sending connection context
0926 15:13:42.434711 (+ 134us) client_negotiation.cc:240] Negotiation successful
0926 15:13:42.434824 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"mutex_wait_us":1,"thread_start_us":283,"threads_started":1}
I20260926 15:13:42.436146 30544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c38d4795bd924553992dbfce24ebc12f" candidate_uuid: "a4b982ef2da84c9082468c2f052957c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bff6b8c127564c09853a756f7d126697" is_pre_election: true
I20260926 15:13:42.436173 30283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c38d4795bd924553992dbfce24ebc12f" candidate_uuid: "a4b982ef2da84c9082468c2f052957c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e149bbb38694cecb33d8bf6b6f729af" is_pre_election: true
I20260926 15:13:42.436702 30283 raft_consensus.cc:2454] T c38d4795bd924553992dbfce24ebc12f P 4e149bbb38694cecb33d8bf6b6f729af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4b982ef2da84c9082468c2f052957c6 in term 0.
I20260926 15:13:42.436691 30544 raft_consensus.cc:2454] T c38d4795bd924553992dbfce24ebc12f P bff6b8c127564c09853a756f7d126697 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4b982ef2da84c9082468c2f052957c6 in term 0.
I20260926 15:13:42.437351 30349 leader_election.cc:302] T c38d4795bd924553992dbfce24ebc12f P a4b982ef2da84c9082468c2f052957c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e149bbb38694cecb33d8bf6b6f729af, a4b982ef2da84c9082468c2f052957c6; no voters:
I20260926 15:13:42.437739 30615 raft_consensus.cc:2781] T c38d4795bd924553992dbfce24ebc12f P a4b982ef2da84c9082468c2f052957c6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 15:13:42.438017 30615 raft_consensus.cc:482] T c38d4795bd924553992dbfce24ebc12f P a4b982ef2da84c9082468c2f052957c6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 15:13:42.438133 30615 raft_consensus.cc:3037] T c38d4795bd924553992dbfce24ebc12f P a4b982ef2da84c9082468c2f052957c6 [term 0 FOLLOWER]: Advancing to term 1
W20260926 15:13:42.438241 30329 tablet.cc:2369] T c38d4795bd924553992dbfce24ebc12f P 4e149bbb38694cecb33d8bf6b6f729af: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 15:13:42.440379 30615 raft_consensus.cc:504] T c38d4795bd924553992dbfce24ebc12f P a4b982ef2da84c9082468c2f052957c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bff6b8c127564c09853a756f7d126697" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 38083 } } peers { permanent_uuid: "a4b982ef2da84c9082468c2f052957c6" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 45131 } } peers { permanent_uuid: "4e149bbb38694cecb33d8bf6b6f729af" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 33681 } }
I20260926 15:13:42.441000 30615 leader_election.cc:288] T c38d4795bd924553992dbfce24ebc12f P a4b982ef2da84c9082468c2f052957c6 [CANDIDATE]: Term 1 election: Requested vote from peers bff6b8c127564c09853a756f7d126697 (127.12.34.195:38083), 4e149bbb38694cecb33d8bf6b6f729af (127.12.34.193:33681)
I20260926 15:13:42.441531 30544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c38d4795bd924553992dbfce24ebc12f" candidate_uuid: "a4b982ef2da84c9082468c2f052957c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bff6b8c127564c09853a756f7d126697"
I20260926 15:13:42.441569 30283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c38d4795bd924553992dbfce24ebc12f" candidate_uuid: "a4b982ef2da84c9082468c2f052957c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e149bbb38694cecb33d8bf6b6f729af"
I20260926 15:13:42.441771 30544 raft_consensus.cc:3037] T c38d4795bd924553992dbfce24ebc12f P bff6b8c127564c09853a756f7d126697 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:13:42.441831 30283 raft_consensus.cc:3037] T c38d4795bd924553992dbfce24ebc12f P 4e149bbb38694cecb33d8bf6b6f729af [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:13:42.444105 30283 raft_consensus.cc:2454] T c38d4795bd924553992dbfce24ebc12f P 4e149bbb38694cecb33d8bf6b6f729af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4b982ef2da84c9082468c2f052957c6 in term 1.
I20260926 15:13:42.444113 30544 raft_consensus.cc:2454] T c38d4795bd924553992dbfce24ebc12f P bff6b8c127564c09853a756f7d126697 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4b982ef2da84c9082468c2f052957c6 in term 1.
I20260926 15:13:42.444494 30349 leader_election.cc:302] T c38d4795bd924553992dbfce24ebc12f P a4b982ef2da84c9082468c2f052957c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a4b982ef2da84c9082468c2f052957c6, bff6b8c127564c09853a756f7d126697; no voters:
I20260926 15:13:42.444804 30615 raft_consensus.cc:2781] T c38d4795bd924553992dbfce24ebc12f P a4b982ef2da84c9082468c2f052957c6 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:13:42.445334 30615 raft_consensus.cc:686] T c38d4795bd924553992dbfce24ebc12f P a4b982ef2da84c9082468c2f052957c6 [term 1 LEADER]: Becoming Leader. State: Replica: a4b982ef2da84c9082468c2f052957c6, State: Running, Role: LEADER
I20260926 15:13:42.445812 30615 consensus_queue.cc:237] T c38d4795bd924553992dbfce24ebc12f P a4b982ef2da84c9082468c2f052957c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bff6b8c127564c09853a756f7d126697" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 38083 } } peers { permanent_uuid: "a4b982ef2da84c9082468c2f052957c6" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 45131 } } peers { permanent_uuid: "4e149bbb38694cecb33d8bf6b6f729af" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 33681 } }
I20260926 15:13:42.450042 30141 catalog_manager.cc:5495] T c38d4795bd924553992dbfce24ebc12f P a4b982ef2da84c9082468c2f052957c6 reported cstate change: term changed from 0 to 1, leader changed from <none> to a4b982ef2da84c9082468c2f052957c6 (127.12.34.194). New cstate: current_term: 1 leader_uuid: "a4b982ef2da84c9082468c2f052957c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bff6b8c127564c09853a756f7d126697" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 38083 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4b982ef2da84c9082468c2f052957c6" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 45131 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e149bbb38694cecb33d8bf6b6f729af" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 33681 } health_report { overall_health: UNKNOWN } } }
I20260926 15:13:42.524420 30604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:42.519451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50930 (local address 127.12.34.194:45131)
0926 15:13:42.519634 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0926 15:13:42.519638 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:13:42.519655 (+ 17us) server_negotiation.cc:407] Connection header received
0926 15:13:42.519861 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:13:42.519865 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:13:42.519966 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:13:42.520093 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 15:13:42.520488 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:42.521249 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:42.522573 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:42.522969 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:42.523067 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:13:42.523462 (+ 395us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 15:13:42.523954 (+ 492us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 15:13:42.524065 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0926 15:13:42.524151 (+ 86us) server_negotiation.cc:299] Negotiation successful
0926 15:13:42.524212 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 15:13:42.532280 30544 raft_consensus.cc:1261] T c38d4795bd924553992dbfce24ebc12f P bff6b8c127564c09853a756f7d126697 [term 1 FOLLOWER]: Refusing update from remote peer a4b982ef2da84c9082468c2f052957c6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:13:42.532373 30283 raft_consensus.cc:1261] T c38d4795bd924553992dbfce24ebc12f P 4e149bbb38694cecb33d8bf6b6f729af [term 1 FOLLOWER]: Refusing update from remote peer a4b982ef2da84c9082468c2f052957c6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:13:42.533252 30615 consensus_queue.cc:1035] T c38d4795bd924553992dbfce24ebc12f P a4b982ef2da84c9082468c2f052957c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bff6b8c127564c09853a756f7d126697" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 38083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:13:42.533550 30618 consensus_queue.cc:1035] T c38d4795bd924553992dbfce24ebc12f P a4b982ef2da84c9082468c2f052957c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e149bbb38694cecb33d8bf6b6f729af" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 33681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:13:42.547935 30622 mvcc.cc:205] Tried to move back new op lower bound from 7333624309882118144 to 7333624309541634048. Current Snapshot: MvccSnapshot[applied={T|T < 7333624309882118144}]
I20260926 15:13:42.549247 30623 mvcc.cc:205] Tried to move back new op lower bound from 7333624309882118144 to 7333624309541634048. Current Snapshot: MvccSnapshot[applied={T|T < 7333624309882118144}]
I20260926 15:13:42.555881 30140 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50204:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 15:13:42.556145 30140 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50204:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 15:13:42.559593 30140 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a3e2ff920f9a467591f4a87a4946a124: Sending DeleteTablet for 3 replicas of tablet c38d4795bd924553992dbfce24ebc12f
I20260926 15:13:42.560637 30394 tablet_service.cc:1514] Processing DeleteTablet for tablet c38d4795bd924553992dbfce24ebc12f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:13:42 UTC) from {username='slave'} at 127.0.0.1:50922
I20260926 15:13:42.560928 30524 tablet_service.cc:1514] Processing DeleteTablet for tablet c38d4795bd924553992dbfce24ebc12f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:13:42 UTC) from {username='slave'} at 127.0.0.1:56866
I20260926 15:13:42.560938 30263 tablet_service.cc:1514] Processing DeleteTablet for tablet c38d4795bd924553992dbfce24ebc12f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:13:42 UTC) from {username='slave'} at 127.0.0.1:51036
I20260926 15:13:42.562732 12427 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 30201
I20260926 15:13:42.573585 30632 tablet_replica.cc:330] T c38d4795bd924553992dbfce24ebc12f P 4e149bbb38694cecb33d8bf6b6f729af: stopping tablet replica
I20260926 15:13:42.574571 30632 raft_consensus.cc:2229] T c38d4795bd924553992dbfce24ebc12f P 4e149bbb38694cecb33d8bf6b6f729af [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 15:13:42.575657 30632 raft_consensus.cc:2258] T c38d4795bd924553992dbfce24ebc12f P 4e149bbb38694cecb33d8bf6b6f729af [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:13:42.578326 30632 ts_tablet_manager.cc:1905] T c38d4795bd924553992dbfce24ebc12f P 4e149bbb38694cecb33d8bf6b6f729af: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:13:42.580003 30630 tablet_replica.cc:330] T c38d4795bd924553992dbfce24ebc12f P a4b982ef2da84c9082468c2f052957c6: stopping tablet replica
I20260926 15:13:42.580848 30631 tablet_replica.cc:330] T c38d4795bd924553992dbfce24ebc12f P bff6b8c127564c09853a756f7d126697: stopping tablet replica
I20260926 15:13:42.581266 30631 raft_consensus.cc:2229] T c38d4795bd924553992dbfce24ebc12f P bff6b8c127564c09853a756f7d126697 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 15:13:42.581768 30631 raft_consensus.cc:2258] T c38d4795bd924553992dbfce24ebc12f P bff6b8c127564c09853a756f7d126697 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:13:42.581394 30630 raft_consensus.cc:2229] T c38d4795bd924553992dbfce24ebc12f P a4b982ef2da84c9082468c2f052957c6 [term 1 LEADER]: Raft consensus shutting down.
I20260926 15:13:42.582614 30630 raft_consensus.cc:2258] T c38d4795bd924553992dbfce24ebc12f P a4b982ef2da84c9082468c2f052957c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:13:42.583666 30631 ts_tablet_manager.cc:1905] T c38d4795bd924553992dbfce24ebc12f P bff6b8c127564c09853a756f7d126697: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:13:42.588008 30630 ts_tablet_manager.cc:1905] T c38d4795bd924553992dbfce24ebc12f P a4b982ef2da84c9082468c2f052957c6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:13:42.588471 30606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:42.564234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51048 (local address 127.12.34.193:33681)
0926 15:13:42.564392 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0926 15:13:42.564398 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:13:42.580274 (+ 15876us) server_negotiation.cc:407] Connection header received
0926 15:13:42.580483 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:13:42.580490 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:13:42.580612 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:13:42.580729 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:13:42.581407 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:42.582495 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:42.584594 (+ 2099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:42.585275 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:42.585371 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:13:42.586819 (+ 1448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:13:42.586830 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:13:42.586848 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:13:42.587566 (+ 718us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:13:42.587623 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:13:42.587647 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:13:42.587728 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 15:13:42.588007 (+ 279us) server_negotiation.cc:299] Negotiation successful
0926 15:13:42.588117 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 15:13:42.589582 30323 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:13:42.591395 30631 ts_tablet_manager.cc:1918] T c38d4795bd924553992dbfce24ebc12f P bff6b8c127564c09853a756f7d126697: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 15:13:42.591588 30631 log.cc:1196] T c38d4795bd924553992dbfce24ebc12f P bff6b8c127564c09853a756f7d126697: Deleting WAL directory at /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790435318278681-12427-0/minicluster-data/ts-2/wal/wals/c38d4795bd924553992dbfce24ebc12f
I20260926 15:13:42.592180 30631 ts_tablet_manager.cc:1939] T c38d4795bd924553992dbfce24ebc12f P bff6b8c127564c09853a756f7d126697: Deleting consensus metadata
I20260926 15:13:42.594419 30127 catalog_manager.cc:4841] TS bff6b8c127564c09853a756f7d126697 (127.12.34.195:38083): tablet c38d4795bd924553992dbfce24ebc12f (table test-table [id=e7c1bbe79108464aa3adb48abb9e33f1]) successfully deleted
I20260926 15:13:42.596966 30630 ts_tablet_manager.cc:1918] T c38d4795bd924553992dbfce24ebc12f P a4b982ef2da84c9082468c2f052957c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 15:13:42.597168 30630 log.cc:1196] T c38d4795bd924553992dbfce24ebc12f P a4b982ef2da84c9082468c2f052957c6: Deleting WAL directory at /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790435318278681-12427-0/minicluster-data/ts-1/wal/wals/c38d4795bd924553992dbfce24ebc12f
I20260926 15:13:42.597728 30630 ts_tablet_manager.cc:1939] T c38d4795bd924553992dbfce24ebc12f P a4b982ef2da84c9082468c2f052957c6: Deleting consensus metadata
I20260926 15:13:42.599738 30126 catalog_manager.cc:4841] TS a4b982ef2da84c9082468c2f052957c6 (127.12.34.194:45131): tablet c38d4795bd924553992dbfce24ebc12f (table test-table [id=e7c1bbe79108464aa3adb48abb9e33f1]) successfully deleted
I20260926 15:13:42.708936 12427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8uoJGv/build/asan/bin/kudu with pid 30201
I20260926 15:13:42.729596 12427 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 30332
W20260926 15:13:42.729662 30127 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.34.193:33681 (error 108)
I20260926 15:13:42.732540 30636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:42.731286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.193:33681 (local address 127.0.0.1:51050)
0926 15:13:42.732268 (+ 982us) negotiation.cc:107] Waiting for socket to connect
0926 15:13:42.732375 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.193:33681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":824,"thread_start_us":750,"threads_started":1}
W20260926 15:13:42.732825 30127 catalog_manager.cc:4584] TS 4e149bbb38694cecb33d8bf6b6f729af (127.12.34.193:33681): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c38d4795bd924553992dbfce24ebc12f: Network error: Client connection negotiation failed: client connection to 127.12.34.193:33681: connect: Connection refused (error 111)
I20260926 15:13:42.745481 30604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:42.730423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50942 (local address 127.12.34.194:45131)
0926 15:13:42.731025 (+ 602us) server_negotiation.cc:206] Beginning negotiation
0926 15:13:42.731032 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:13:42.731050 (+ 18us) server_negotiation.cc:407] Connection header received
0926 15:13:42.731116 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:13:42.731121 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:13:42.731228 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:13:42.731332 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:13:42.731909 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:42.733127 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:42.734364 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:42.735035 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:42.735156 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:13:42.743709 (+ 8553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:13:42.743718 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:13:42.743732 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:13:42.744417 (+ 685us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:13:42.744471 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:13:42.744493 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:13:42.744583 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 15:13:42.744809 (+ 226us) server_negotiation.cc:299] Negotiation successful
0926 15:13:42.744912 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":466}
I20260926 15:13:42.746328 30454 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:13:42.786441 30636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:42.785937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.193:33681 (local address 127.0.0.1:51062)
0926 15:13:42.786184 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0926 15:13:42.786278 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.193:33681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260926 15:13:42.787612 30636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:42.787188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.193:33681 (local address 127.0.0.1:51072)
0926 15:13:42.787360 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0926 15:13:42.787449 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.193:33681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260926 15:13:42.848330 30636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:42.847772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.193:33681 (local address 127.0.0.1:51084)
0926 15:13:42.848034 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0926 15:13:42.848145 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.193:33681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260926 15:13:42.849385 30636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:42.848981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.193:33681 (local address 127.0.0.1:51100)
0926 15:13:42.849163 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0926 15:13:42.849239 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.193:33681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260926 15:13:42.857896 12427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8uoJGv/build/asan/bin/kudu with pid 30332
I20260926 15:13:42.877202 12427 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 30462
I20260926 15:13:42.883841 30603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:42.877973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56870 (local address 127.12.34.195:38083)
0926 15:13:42.878142 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0926 15:13:42.878147 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:13:42.878166 (+ 19us) server_negotiation.cc:407] Connection header received
0926 15:13:42.878317 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:13:42.878320 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:13:42.878411 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:13:42.878495 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:13:42.878910 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:42.879592 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:42.880876 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:42.881349 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:42.881442 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:13:42.882543 (+ 1101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:13:42.882551 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:13:42.882563 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:13:42.883112 (+ 549us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:13:42.883156 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:13:42.883180 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:13:42.883264 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 15:13:42.883409 (+ 145us) server_negotiation.cc:299] Negotiation successful
0926 15:13:42.883489 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 15:13:42.884531 30584 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:13:42.930159 30636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:42.929631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.193:33681 (local address 127.0.0.1:51116)
0926 15:13:42.929873 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0926 15:13:42.929990 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.193:33681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
I20260926 15:13:42.931246 30636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:42.930851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.193:33681 (local address 127.0.0.1:51122)
0926 15:13:42.931051 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0926 15:13:42.931117 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.193:33681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260926 15:13:42.973342 12427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8uoJGv/build/asan/bin/kudu with pid 30462
I20260926 15:13:42.989852 12427 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 30110
I20260926 15:13:42.990962 30171 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:13:43.100378 12427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8uoJGv/build/asan/bin/kudu with pid 30110
2026-09-26T15:13:43Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (2956 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435318278681-12427-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 15:13:43 dist-test-slave-865v krb5kdc[30642](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 15:13:43 dist-test-slave-865v krb5kdc[30642](info): set up 2 sockets
Sep 26 15:13:43 dist-test-slave-865v krb5kdc[30642](info): commencing operation
krb5kdc: starting...
W20260926 15:13:44.217271 12427 mini_kdc.cc:121] Time spent starting KDC: real 1.076s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435318278681-12427-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435318278681-12427-0/krb5kdc/test-user.keytab.
Sep 26 15:13:44 dist-test-slave-865v krb5kdc[30642](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435624, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T15:13:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T15:13:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.34.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.34.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.34.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.254.keytab.
Entry for principal kudu/127.12.34.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.254.keytab.
I20260926 15:13:44.456310 12427 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8uoJGv/build/asan/bin/kudu
/tmp/dist-test-task8uoJGv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435318278681-12427-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435318278681-12427-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435318278681-12427-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.34.254:45895
--webserver_interface=127.12.34.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.254.keytab
--principal=kudu/127.12.34.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.34.212:40593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.34.254:45895
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435318278681-12427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435318278681-12427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435318278681-12427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:13:44.776674 30655 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:13:44.776945 30655 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:13:44.777072 30655 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:13:44.784343 30655 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:13:44.784451 30655 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:13:44.784536 30655 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:13:44.784646 30655 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:13:44.784708 30655 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:13:44.794672 30655 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.34.212:40593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435318278681-12427-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435318278681-12427-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.34.254:45895
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.254.keytab
--principal=kudu/127.12.34.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.34.254:45895
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.34.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435318278681-12427-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:13:44.796036 30655 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:13:44.797387 30655 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:13:44.811169 30661 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:13:44.811251 30660 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:13:44.811251 30663 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:13:44.812569 30655 server_base.cc:1034] running on GCE node
I20260926 15:13:44.813609 30655 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:13:44.815141 30655 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:13:44.816536 30655 hybrid_clock.cc:648] HybridClock initialized: now 1790435624816427 us; error 69 us; skew 500 ppm
I20260926 15:13:44.817276 30655 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:13:44 dist-test-slave-865v krb5kdc[30642](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435624, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:13:44.821019 30655 init.cc:377] Logged in from keytab as kudu/127.12.34.254@KRBTEST.COM (short username kudu)
I20260926 15:13:44.822865 30655 webserver.cc:460] Webserver started at http://127.12.34.254:45845/ using document root <none> and password file <none>
I20260926 15:13:44.823535 30655 fs_manager.cc:359] Metadata directory not provided
I20260926 15:13:44.823735 30655 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:13:44.824020 30655 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:13:44.827111 30655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435318278681-12427-0/minicluster-data/master-0/data/instance:
uuid: "8b45f55caaa84dcfa1ea3e2846646453"
format_stamp: "Formatted at 2026-09-26 15:13:44 on dist-test-slave-865v"
server_key: "af589c95172338c86bc0600ce38495ec"
server_key_iv: "b4e3dc565d6ced9073363978bd7c8a68"
server_key_version: "encryptionkey@0"
I20260926 15:13:44.827852 30655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435318278681-12427-0/minicluster-data/master-0/wal/instance:
uuid: "8b45f55caaa84dcfa1ea3e2846646453"
format_stamp: "Formatted at 2026-09-26 15:13:44 on dist-test-slave-865v"
server_key: "af589c95172338c86bc0600ce38495ec"
server_key_iv: "b4e3dc565d6ced9073363978bd7c8a68"
server_key_version: "encryptionkey@0"
I20260926 15:13:44.835419 30655 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20260926 15:13:44.840240 30670 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:13:44.842312 30655 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260926 15:13:44.842491 30655 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435318278681-12427-0/minicluster-data/master-0/data,/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435318278681-12427-0/minicluster-data/master-0/wal
uuid: "8b45f55caaa84dcfa1ea3e2846646453"
format_stamp: "Formatted at 2026-09-26 15:13:44 on dist-test-slave-865v"
server_key: "af589c95172338c86bc0600ce38495ec"
server_key_iv: "b4e3dc565d6ced9073363978bd7c8a68"
server_key_version: "encryptionkey@0"
I20260926 15:13:44.842684 30655 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435318278681-12427-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435318278681-12427-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435318278681-12427-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:13:44.866714 30655 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:13:44.871112 30655 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:13:44.871351 30655 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:13:44.890067 30655 rpc_server.cc:307] RPC server started. Bound to: 127.12.34.254:45895
I20260926 15:13:44.890106 30721 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.34.254:45895 every 8 connection(s)
I20260926 15:13:44.891415 30655 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
I20260926 15:13:44.895048 30722 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:13:44.898798 12427 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8uoJGv/build/asan/bin/kudu as pid 30655
I20260926 15:13:44.898983 12427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435318278681-12427-0/minicluster-data/master-0/wal/instance
I20260926 15:13:44.899556 12427 external_mini_cluster.cc:1428] Setting key 8572b6bf3d0912e241ea4a26c9aebfc6
I20260926 15:13:44.907531 30722 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b45f55caaa84dcfa1ea3e2846646453: Bootstrap starting.
I20260926 15:13:44.910590 30722 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8b45f55caaa84dcfa1ea3e2846646453: Neither blocks nor log segments found. Creating new log.
I20260926 15:13:44.911566 30722 log.cc:824] T 00000000000000000000000000000000 P 8b45f55caaa84dcfa1ea3e2846646453: Log is configured to *not* fsync() on all Append() calls
Sep 26 15:13:44 dist-test-slave-865v krb5kdc[30642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435624, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.34.254@KRBTEST.COM
I20260926 15:13:44.914636 30722 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b45f55caaa84dcfa1ea3e2846646453: No bootstrap required, opened a new log
I20260926 15:13:44.921908 30725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:44.901566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51966 (local address 127.12.34.254:45895)
0926 15:13:44.902349 (+ 783us) server_negotiation.cc:206] Beginning negotiation
0926 15:13:44.902362 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 15:13:44.902404 (+ 42us) server_negotiation.cc:407] Connection header received
0926 15:13:44.904142 (+ 1738us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:13:44.904164 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:13:44.904567 (+ 403us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:13:44.904961 (+ 394us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:13:44.906509 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:44.907861 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:44.909182 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:44.909836 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:44.910031 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:13:44.914441 (+ 4410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:13:44.914449 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:13:44.914463 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:13:44.914497 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:13:44.917748 (+ 3251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:13:44.918587 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:13:44.918593 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:13:44.918599 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:13:44.918767 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:13:44.919197 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:13:44.919200 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:13:44.919201 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:13:44.919706 (+ 505us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:13:44.919947 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0926 15:13:44.920428 (+ 481us) server_negotiation.cc:299] Negotiation successful
0926 15:13:44.920808 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":562,"thread_start_us":348,"threads_started":1}
I20260926 15:13:44.923488 30722 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8b45f55caaa84dcfa1ea3e2846646453 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b45f55caaa84dcfa1ea3e2846646453" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 45895 } }
I20260926 15:13:44.923874 30722 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8b45f55caaa84dcfa1ea3e2846646453 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:13:44.923997 30722 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8b45f55caaa84dcfa1ea3e2846646453 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b45f55caaa84dcfa1ea3e2846646453, State: Initialized, Role: FOLLOWER
I20260926 15:13:44.924587 30722 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8b45f55caaa84dcfa1ea3e2846646453 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b45f55caaa84dcfa1ea3e2846646453" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 45895 } }
I20260926 15:13:44.924834 30722 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8b45f55caaa84dcfa1ea3e2846646453 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 15:13:44.924976 30722 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8b45f55caaa84dcfa1ea3e2846646453 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 15:13:44.925153 30722 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8b45f55caaa84dcfa1ea3e2846646453 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:13:44.927263 30722 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8b45f55caaa84dcfa1ea3e2846646453 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b45f55caaa84dcfa1ea3e2846646453" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 45895 } }
I20260926 15:13:44.927722 30722 leader_election.cc:302] T 00000000000000000000000000000000 P 8b45f55caaa84dcfa1ea3e2846646453 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8b45f55caaa84dcfa1ea3e2846646453; no voters:
I20260926 15:13:44.928292 30722 leader_election.cc:288] T 00000000000000000000000000000000 P 8b45f55caaa84dcfa1ea3e2846646453 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 15:13:44.928526 30727 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8b45f55caaa84dcfa1ea3e2846646453 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:13:44.929352 30727 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8b45f55caaa84dcfa1ea3e2846646453 [term 1 LEADER]: Becoming Leader. State: Replica: 8b45f55caaa84dcfa1ea3e2846646453, State: Running, Role: LEADER
I20260926 15:13:44.929890 30727 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8b45f55caaa84dcfa1ea3e2846646453 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b45f55caaa84dcfa1ea3e2846646453" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 45895 } }
I20260926 15:13:44.930397 30722 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8b45f55caaa84dcfa1ea3e2846646453 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:13:44.932984 30729 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b45f55caaa84dcfa1ea3e2846646453 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8b45f55caaa84dcfa1ea3e2846646453. Latest consensus state: current_term: 1 leader_uuid: "8b45f55caaa84dcfa1ea3e2846646453" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b45f55caaa84dcfa1ea3e2846646453" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 45895 } } }
I20260926 15:13:44.932989 30728 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b45f55caaa84dcfa1ea3e2846646453 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8b45f55caaa84dcfa1ea3e2846646453" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b45f55caaa84dcfa1ea3e2846646453" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 45895 } } }
I20260926 15:13:44.933372 30729 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b45f55caaa84dcfa1ea3e2846646453 [sys.catalog]: This master's current role is: LEADER
I20260926 15:13:44.933420 30728 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b45f55caaa84dcfa1ea3e2846646453 [sys.catalog]: This master's current role is: LEADER
I20260926 15:13:44.935441 30734 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:13:44.939720 30734 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 15:13:44.947793 30734 catalog_manager.cc:1347] Generated new cluster ID: 918b7fd392aa445b974ec919e3594d40
I20260926 15:13:44.947916 30734 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 15:13:44.961279 30734 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 15:13:44.963060 30734 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:13:44.972393 30734 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8b45f55caaa84dcfa1ea3e2846646453: Generated new TSK 0
I20260926 15:13:44.973361 30734 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.34.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.34.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.34.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.193.keytab.
Entry for principal kudu/127.12.34.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.193.keytab.
I20260926 15:13:45.033241 12427 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8uoJGv/build/asan/bin/kudu
/tmp/dist-test-task8uoJGv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435318278681-12427-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435318278681-12427-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435318278681-12427-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435318278681-12427-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.34.193:0
--local_ip_for_outbound_sockets=127.12.34.193
--webserver_interface=127.12.34.193
--webserver_port=0
--tserver_master_addrs=127.12.34.254:45895
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.193.keytab
--principal=kudu/127.12.34.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.34.212:40593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435318278681-12427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435318278681-12427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435318278681-12427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:13:45.359928 30748 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:13:45.360172 30748 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:13:45.360262 30748 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:13:45.367493 30748 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:13:45.367604 30748 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:13:45.367717 30748 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.34.193
I20260926 15:13:45.377239 30748 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.34.212:40593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435318278681-12427-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435318278681-12427-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.193.keytab
--principal=kudu/127.12.34.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.34.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435318278681-12427-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.34.193
--webserver_port=0
--tserver_master_addrs=127.12.34.254:45895
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.34.193
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435318278681-12427-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:13:45.378386 30748 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:13:45.379567 30748 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:13:45.392093 30756 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:13:45.392164 30753 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:13:45.392143 30754 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:13:45.393476 30748 server_base.cc:1034] running on GCE node
I20260926 15:13:45.394073 30748 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:13:45.394994 30748 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:13:45.396219 30748 hybrid_clock.cc:648] HybridClock initialized: now 1790435625396175 us; error 51 us; skew 500 ppm
I20260926 15:13:45.396528 30748 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:13:45 dist-test-slave-865v krb5kdc[30642](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435625, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:13:45.399613 30748 init.cc:377] Logged in from keytab as kudu/127.12.34.193@KRBTEST.COM (short username kudu)
I20260926 15:13:45.400909 30748 webserver.cc:460] Webserver started at http://127.12.34.193:38077/ using document root <none> and password file <none>
I20260926 15:13:45.401582 30748 fs_manager.cc:359] Metadata directory not provided
I20260926 15:13:45.401698 30748 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:13:45.401937 30748 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:13:45.404724 30748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435318278681-12427-0/minicluster-data/ts-0/data/instance:
uuid: "d8eea60de04541fe8aef7c71f01da2d4"
format_stamp: "Formatted at 2026-09-26 15:13:45 on dist-test-slave-865v"
server_key: "7a5551b4c722e70afd1e4ed9c2145e7e"
server_key_iv: "389f5fe7e723d44d6ee80bd6417c6a00"
server_key_version: "encryptionkey@0"
I20260926 15:13:45.405503 30748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435318278681-12427-0/minicluster-data/ts-0/wal/instance:
uuid: "d8eea60de04541fe8aef7c71f01da2d4"
format_stamp: "Formatted at 2026-09-26 15:13:45 on dist-test-slave-865v"
server_key: "7a5551b4c722e70afd1e4ed9c2145e7e"
server_key_iv: "389f5fe7e723d44d6ee80bd6417c6a00"
server_key_version: "encryptionkey@0"
I20260926 15:13:45.410435 30748 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260926 15:13:45.413666 30763 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:13:45.414896 30748 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260926 15:13:45.415016 30748 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435318278681-12427-0/minicluster-data/ts-0/data,/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435318278681-12427-0/minicluster-data/ts-0/wal
uuid: "d8eea60de04541fe8aef7c71f01da2d4"
format_stamp: "Formatted at 2026-09-26 15:13:45 on dist-test-slave-865v"
server_key: "7a5551b4c722e70afd1e4ed9c2145e7e"
server_key_iv: "389f5fe7e723d44d6ee80bd6417c6a00"
server_key_version: "encryptionkey@0"
I20260926 15:13:45.415177 30748 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435318278681-12427-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435318278681-12427-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435318278681-12427-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:13:45.427008 30748 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:13:45.430635 30748 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:13:45.430866 30748 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:13:45.431739 30748 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:13:45.433127 30748 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:13:45.433239 30748 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:13:45.433337 30748 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:13:45.433382 30748 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:13:45.466084 30748 rpc_server.cc:307] RPC server started. Bound to: 127.12.34.193:46527
I20260926 15:13:45.466132 30875 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.34.193:46527 every 8 connection(s)
I20260926 15:13:45.467370 30748 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435318278681-12427-0/minicluster-data/ts-0/data/info.pb
I20260926 15:13:45.476269 12427 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8uoJGv/build/asan/bin/kudu as pid 30748
I20260926 15:13:45.476464 12427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435318278681-12427-0/minicluster-data/ts-0/wal/instance
I20260926 15:13:45.477035 12427 external_mini_cluster.cc:1428] Setting key 507f7b9eed08cd20d73464f3e83e7454
Sep 26 15:13:45 dist-test-slave-865v krb5kdc[30642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435625, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.193@KRBTEST.COM for kudu/127.12.34.254@KRBTEST.COM
I20260926 15:13:45.488377 30878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:45.469970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.193:58695 (local address 127.12.34.254:45895)
0926 15:13:45.470474 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0926 15:13:45.470479 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:13:45.471373 (+ 894us) server_negotiation.cc:407] Connection header received
0926 15:13:45.473257 (+ 1884us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:13:45.473262 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:13:45.473349 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:13:45.473465 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:13:45.474880 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:45.475571 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:45.477797 (+ 2226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:45.478305 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:45.478428 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:13:45.481838 (+ 3410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:13:45.481847 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:13:45.481854 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:13:45.481897 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:13:45.485365 (+ 3468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:13:45.486044 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:13:45.486052 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:13:45.486058 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:13:45.486206 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:13:45.486599 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:13:45.486607 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:13:45.486613 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:13:45.486998 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:13:45.487153 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0926 15:13:45.487968 (+ 815us) server_negotiation.cc:299] Negotiation successful
0926 15:13:45.488124 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":269,"threads_started":1}
I20260926 15:13:45.489807 30879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:45.470498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:45895 (local address 127.12.34.193:58695)
0926 15:13:45.471201 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0926 15:13:45.471250 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 15:13:45.472946 (+ 1696us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:13:45.473690 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:13:45.473697 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:13:45.474164 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:13:45.474686 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:13:45.474702 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:13:45.475718 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:13:45.475725 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:13:45.477647 (+ 1922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:13:45.477659 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:13:45.478521 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:13:45.478526 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:13:45.478779 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:13:45.479503 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:13:45.479525 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:13:45.481624 (+ 2099us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:13:45.485498 (+ 3874us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:13:45.485505 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:13:45.485516 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:13:45.485914 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:13:45.486307 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:13:45.486309 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:13:45.486312 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:13:45.486486 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:13:45.487120 (+ 634us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:13:45.487125 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:13:45.487572 (+ 447us) client_negotiation.cc:769] Sending connection context
0926 15:13:45.487871 (+ 299us) client_negotiation.cc:240] Negotiation successful
0926 15:13:45.488436 (+ 565us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":285,"threads_started":1}
I20260926 15:13:45.491273 30876 heartbeater.cc:347] Connected to a master server at 127.12.34.254:45895
I20260926 15:13:45.491560 30876 heartbeater.cc:464] Registering TS with master...
I20260926 15:13:45.492238 30876 heartbeater.cc:511] Master 127.12.34.254:45895 requested a full tablet report, sending...
I20260926 15:13:45.494210 30687 ts_manager.cc:194] Registered new tserver with Master: d8eea60de04541fe8aef7c71f01da2d4 (127.12.34.193:46527)
I20260926 15:13:45.495579 30687 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.34.193@KRBTEST.COM'} at 127.12.34.193:58695
WARNING: no policy specified for kudu/127.12.34.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.34.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.34.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.194.keytab.
Entry for principal kudu/127.12.34.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.194.keytab.
I20260926 15:13:45.532644 12427 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8uoJGv/build/asan/bin/kudu
/tmp/dist-test-task8uoJGv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435318278681-12427-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435318278681-12427-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435318278681-12427-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435318278681-12427-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.34.194:0
--local_ip_for_outbound_sockets=127.12.34.194
--webserver_interface=127.12.34.194
--webserver_port=0
--tserver_master_addrs=127.12.34.254:45895
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.194.keytab
--principal=kudu/127.12.34.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.34.212:40593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435318278681-12427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435318278681-12427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435318278681-12427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:13:45.848560 30882 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:13:45.848871 30882 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:13:45.849036 30882 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:13:45.856405 30882 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:13:45.856550 30882 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:13:45.856725 30882 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.34.194
I20260926 15:13:45.866335 30882 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.34.212:40593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435318278681-12427-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435318278681-12427-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.194.keytab
--principal=kudu/127.12.34.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.34.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435318278681-12427-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.34.194
--webserver_port=0
--tserver_master_addrs=127.12.34.254:45895
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.34.194
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435318278681-12427-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:13:45.867602 30882 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:13:45.868868 30882 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:13:45.880617 30887 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:13:45.881367 30888 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:13:45.882149 30890 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:13:45.883016 30882 server_base.cc:1034] running on GCE node
I20260926 15:13:45.883579 30882 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:13:45.884450 30882 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:13:45.885674 30882 hybrid_clock.cc:648] HybridClock initialized: now 1790435625885630 us; error 53 us; skew 500 ppm
I20260926 15:13:45.885970 30882 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:13:45 dist-test-slave-865v krb5kdc[30642](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435625, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:13:45.889019 30882 init.cc:377] Logged in from keytab as kudu/127.12.34.194@KRBTEST.COM (short username kudu)
I20260926 15:13:45.890574 30882 webserver.cc:460] Webserver started at http://127.12.34.194:36623/ using document root <none> and password file <none>
I20260926 15:13:45.891237 30882 fs_manager.cc:359] Metadata directory not provided
I20260926 15:13:45.891330 30882 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:13:45.891664 30882 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:13:45.894485 30882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435318278681-12427-0/minicluster-data/ts-1/data/instance:
uuid: "df798f79f34b4bb7bb0a4bc242fe0ff8"
format_stamp: "Formatted at 2026-09-26 15:13:45 on dist-test-slave-865v"
server_key: "f4bc29a62fc132fb42f40b3cdbada325"
server_key_iv: "0f730c33f32887aad4459d376a0f1df4"
server_key_version: "encryptionkey@0"
I20260926 15:13:45.895236 30882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435318278681-12427-0/minicluster-data/ts-1/wal/instance:
uuid: "df798f79f34b4bb7bb0a4bc242fe0ff8"
format_stamp: "Formatted at 2026-09-26 15:13:45 on dist-test-slave-865v"
server_key: "f4bc29a62fc132fb42f40b3cdbada325"
server_key_iv: "0f730c33f32887aad4459d376a0f1df4"
server_key_version: "encryptionkey@0"
I20260926 15:13:45.900765 30882 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260926 15:13:45.904428 30897 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:13:45.906080 30882 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 15:13:45.906277 30882 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435318278681-12427-0/minicluster-data/ts-1/data,/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435318278681-12427-0/minicluster-data/ts-1/wal
uuid: "df798f79f34b4bb7bb0a4bc242fe0ff8"
format_stamp: "Formatted at 2026-09-26 15:13:45 on dist-test-slave-865v"
server_key: "f4bc29a62fc132fb42f40b3cdbada325"
server_key_iv: "0f730c33f32887aad4459d376a0f1df4"
server_key_version: "encryptionkey@0"
I20260926 15:13:45.906479 30882 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435318278681-12427-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435318278681-12427-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435318278681-12427-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:13:45.922803 30882 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:13:45.927378 30882 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:13:45.927693 30882 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:13:45.928660 30882 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:13:45.930027 30882 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:13:45.930115 30882 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:13:45.930307 30882 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:13:45.930367 30882 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:13:45.966416 30882 rpc_server.cc:307] RPC server started. Bound to: 127.12.34.194:43553
I20260926 15:13:45.966490 31009 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.34.194:43553 every 8 connection(s)
I20260926 15:13:45.967760 30882 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435318278681-12427-0/minicluster-data/ts-1/data/info.pb
I20260926 15:13:45.974623 12427 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8uoJGv/build/asan/bin/kudu as pid 30882
I20260926 15:13:45.974817 12427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435318278681-12427-0/minicluster-data/ts-1/wal/instance
I20260926 15:13:45.975235 12427 external_mini_cluster.cc:1428] Setting key de96038c05eb18d168de2116f187890f
Sep 26 15:13:45 dist-test-slave-865v krb5kdc[30642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435625, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.194@KRBTEST.COM for kudu/127.12.34.254@KRBTEST.COM
I20260926 15:13:45.988021 30878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:45.970312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.194:45649 (local address 127.12.34.254:45895)
0926 15:13:45.970505 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0926 15:13:45.970510 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:13:45.971578 (+ 1068us) server_negotiation.cc:407] Connection header received
0926 15:13:45.973586 (+ 2008us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:13:45.973592 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:13:45.973687 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:13:45.973817 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:13:45.975460 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:45.976398 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:45.977992 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:45.978398 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:45.978533 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:13:45.982170 (+ 3637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:13:45.982175 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:13:45.982179 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:13:45.982213 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:13:45.984837 (+ 2624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:13:45.985620 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:13:45.985625 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:13:45.985628 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:13:45.985748 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:13:45.986201 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:13:45.986210 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:13:45.986213 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:13:45.986465 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:13:45.986615 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0926 15:13:45.987315 (+ 700us) server_negotiation.cc:299] Negotiation successful
0926 15:13:45.987573 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 15:13:45.988760 31012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:45.970704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:45895 (local address 127.12.34.194:45649)
0926 15:13:45.971414 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0926 15:13:45.971460 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 15:13:45.973310 (+ 1850us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:13:45.974056 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:13:45.974064 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:13:45.974584 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:13:45.975257 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:13:45.975275 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:13:45.976557 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:13:45.976563 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:13:45.977845 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:13:45.977858 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:13:45.978615 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:13:45.978622 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:13:45.978842 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:13:45.979519 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:13:45.979541 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:13:45.981924 (+ 2383us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:13:45.984983 (+ 3059us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:13:45.984993 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:13:45.985008 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:13:45.985488 (+ 480us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:13:45.985883 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:13:45.985890 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:13:45.985893 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:13:45.986080 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:13:45.986584 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:13:45.986592 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:13:45.986917 (+ 325us) client_negotiation.cc:769] Sending connection context
0926 15:13:45.987169 (+ 252us) client_negotiation.cc:240] Negotiation successful
0926 15:13:45.987622 (+ 453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":271,"threads_started":1}
I20260926 15:13:45.990221 31010 heartbeater.cc:347] Connected to a master server at 127.12.34.254:45895
I20260926 15:13:45.990510 31010 heartbeater.cc:464] Registering TS with master...
I20260926 15:13:45.991178 31010 heartbeater.cc:511] Master 127.12.34.254:45895 requested a full tablet report, sending...
I20260926 15:13:45.992578 30687 ts_manager.cc:194] Registered new tserver with Master: df798f79f34b4bb7bb0a4bc242fe0ff8 (127.12.34.194:43553)
I20260926 15:13:45.993813 30687 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.34.194@KRBTEST.COM'} at 127.12.34.194:45649
WARNING: no policy specified for kudu/127.12.34.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.34.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.34.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.195.keytab.
Entry for principal kudu/127.12.34.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.195.keytab.
I20260926 15:13:46.032142 12427 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8uoJGv/build/asan/bin/kudu
/tmp/dist-test-task8uoJGv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435318278681-12427-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435318278681-12427-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435318278681-12427-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435318278681-12427-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.34.195:0
--local_ip_for_outbound_sockets=127.12.34.195
--webserver_interface=127.12.34.195
--webserver_port=0
--tserver_master_addrs=127.12.34.254:45895
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.195.keytab
--principal=kudu/127.12.34.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.34.212:40593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435318278681-12427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435318278681-12427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435318278681-12427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:13:46.353507 31015 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:13:46.353935 31015 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:13:46.354158 31015 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:13:46.361609 31015 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:13:46.361699 31015 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:13:46.361806 31015 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.34.195
I20260926 15:13:46.371459 31015 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.34.212:40593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435318278681-12427-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435318278681-12427-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.195.keytab
--principal=kudu/127.12.34.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.34.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435318278681-12427-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.34.195
--webserver_port=0
--tserver_master_addrs=127.12.34.254:45895
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.34.195
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435318278681-12427-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:13:46.372581 31015 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:13:46.373873 31015 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:13:46.386087 31020 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:13:46.386344 31021 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:13:46.386749 31023 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:13:46.387549 31015 server_base.cc:1034] running on GCE node
I20260926 15:13:46.388154 31015 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:13:46.389405 31015 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:13:46.390645 31015 hybrid_clock.cc:648] HybridClock initialized: now 1790435626390600 us; error 64 us; skew 500 ppm
I20260926 15:13:46.390961 31015 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:13:46 dist-test-slave-865v krb5kdc[30642](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435626, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:13:46.394202 31015 init.cc:377] Logged in from keytab as kudu/127.12.34.195@KRBTEST.COM (short username kudu)
I20260926 15:13:46.395649 31015 webserver.cc:460] Webserver started at http://127.12.34.195:43083/ using document root <none> and password file <none>
I20260926 15:13:46.396349 31015 fs_manager.cc:359] Metadata directory not provided
I20260926 15:13:46.396489 31015 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:13:46.396792 31015 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:13:46.399641 31015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435318278681-12427-0/minicluster-data/ts-2/data/instance:
uuid: "6486c17b21eb4251840e1d8332864e10"
format_stamp: "Formatted at 2026-09-26 15:13:46 on dist-test-slave-865v"
server_key: "18e0feb6b82e7fc276bd773c02e8bf7d"
server_key_iv: "865fcde49cc9c6ecc34b0059baef1784"
server_key_version: "encryptionkey@0"
I20260926 15:13:46.400384 31015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435318278681-12427-0/minicluster-data/ts-2/wal/instance:
uuid: "6486c17b21eb4251840e1d8332864e10"
format_stamp: "Formatted at 2026-09-26 15:13:46 on dist-test-slave-865v"
server_key: "18e0feb6b82e7fc276bd773c02e8bf7d"
server_key_iv: "865fcde49cc9c6ecc34b0059baef1784"
server_key_version: "encryptionkey@0"
I20260926 15:13:46.405524 31015 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 15:13:46.409363 31030 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:13:46.410665 31015 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 15:13:46.410836 31015 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435318278681-12427-0/minicluster-data/ts-2/data,/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435318278681-12427-0/minicluster-data/ts-2/wal
uuid: "6486c17b21eb4251840e1d8332864e10"
format_stamp: "Formatted at 2026-09-26 15:13:46 on dist-test-slave-865v"
server_key: "18e0feb6b82e7fc276bd773c02e8bf7d"
server_key_iv: "865fcde49cc9c6ecc34b0059baef1784"
server_key_version: "encryptionkey@0"
I20260926 15:13:46.411027 31015 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435318278681-12427-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435318278681-12427-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435318278681-12427-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:13:46.422703 31015 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:13:46.426406 31015 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:13:46.426693 31015 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:13:46.427582 31015 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:13:46.428803 31015 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:13:46.428902 31015 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:13:46.429047 31015 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:13:46.429154 31015 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260926 15:13:46.464905 31015 rpc_server.cc:307] RPC server started. Bound to: 127.12.34.195:41203
I20260926 15:13:46.464946 31142 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.34.195:41203 every 8 connection(s)
I20260926 15:13:46.466524 31015 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435318278681-12427-0/minicluster-data/ts-2/data/info.pb
I20260926 15:13:46.474692 12427 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8uoJGv/build/asan/bin/kudu as pid 31015
I20260926 15:13:46.474898 12427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435318278681-12427-0/minicluster-data/ts-2/wal/instance
I20260926 15:13:46.475337 12427 external_mini_cluster.cc:1428] Setting key 32cad49c920455e85c975d1628c29557
Sep 26 15:13:46 dist-test-slave-865v krb5kdc[30642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435626, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.195@KRBTEST.COM for kudu/127.12.34.254@KRBTEST.COM
I20260926 15:13:46.487789 30878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:46.469861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.195:54791 (local address 127.12.34.254:45895)
0926 15:13:46.470031 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0926 15:13:46.470035 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:13:46.471237 (+ 1202us) server_negotiation.cc:407] Connection header received
0926 15:13:46.473073 (+ 1836us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:13:46.473111 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:13:46.473196 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:13:46.473317 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:13:46.474739 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:46.475618 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:46.477353 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:46.477791 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:46.477921 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:13:46.482096 (+ 4175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:13:46.482101 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:13:46.482104 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:13:46.482136 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:13:46.484798 (+ 2662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:13:46.485719 (+ 921us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:13:46.485724 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:13:46.485726 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:13:46.485868 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:13:46.486298 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:13:46.486301 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:13:46.486303 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:13:46.486560 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:13:46.486685 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0926 15:13:46.487389 (+ 704us) server_negotiation.cc:299] Negotiation successful
0926 15:13:46.487548 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 15:13:46.488749 31145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:46.470310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:45895 (local address 127.12.34.195:54791)
0926 15:13:46.471070 (+ 760us) negotiation.cc:107] Waiting for socket to connect
0926 15:13:46.471108 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 15:13:46.472782 (+ 1674us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:13:46.473541 (+ 759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:13:46.473548 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:13:46.474039 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:13:46.474568 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:13:46.474585 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:13:46.475767 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:13:46.475774 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:13:46.477212 (+ 1438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:13:46.477233 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:13:46.478177 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:13:46.478183 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:13:46.478380 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:13:46.479094 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:13:46.479117 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:13:46.481810 (+ 2693us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:13:46.484971 (+ 3161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:13:46.484987 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:13:46.485001 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:13:46.485569 (+ 568us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:13:46.485982 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:13:46.485985 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:13:46.485988 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:13:46.486167 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:13:46.486692 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:13:46.486700 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:13:46.487020 (+ 320us) client_negotiation.cc:769] Sending connection context
0926 15:13:46.487258 (+ 238us) client_negotiation.cc:240] Negotiation successful
0926 15:13:46.487622 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":353,"threads_started":1}
I20260926 15:13:46.490579 31143 heartbeater.cc:347] Connected to a master server at 127.12.34.254:45895
I20260926 15:13:46.490892 31143 heartbeater.cc:464] Registering TS with master...
I20260926 15:13:46.491568 31143 heartbeater.cc:511] Master 127.12.34.254:45895 requested a full tablet report, sending...
I20260926 15:13:46.492985 30687 ts_manager.cc:194] Registered new tserver with Master: 6486c17b21eb4251840e1d8332864e10 (127.12.34.195:41203)
I20260926 15:13:46.493818 30687 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.34.195@KRBTEST.COM'} at 127.12.34.195:54791
I20260926 15:13:46.498348 30876 heartbeater.cc:503] Master 127.12.34.254:45895 was elected leader, sending a full tablet report...
I20260926 15:13:46.503289 12427 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 15:13:46.520840 30878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:46.507333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51976 (local address 127.12.34.254:45895)
0926 15:13:46.507494 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0926 15:13:46.507498 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:13:46.507962 (+ 464us) server_negotiation.cc:407] Connection header received
0926 15:13:46.508130 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:13:46.508135 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:13:46.508202 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:13:46.508301 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:13:46.509743 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:46.511066 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:46.512348 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:46.512753 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:46.512885 (+ 132us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:13:46.515747 (+ 2862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:13:46.515752 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:13:46.515755 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:13:46.515785 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:13:46.518305 (+ 2520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:13:46.519026 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:13:46.519030 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:13:46.519032 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:13:46.519152 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:13:46.519592 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:13:46.519596 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:13:46.519598 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:13:46.519879 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:13:46.520081 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0926 15:13:46.520476 (+ 395us) server_negotiation.cc:299] Negotiation successful
0926 15:13:46.520605 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 15:13:46.524657 30687 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51976:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 15:13:46.531548 30687 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 15:13:46.555015 31159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:46.545916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.193:46527 (local address 127.0.0.1:58118)
0926 15:13:46.547217 (+ 1301us) negotiation.cc:107] Waiting for socket to connect
0926 15:13:46.547238 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 15:13:46.547391 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:13:46.547994 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:13:46.547998 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:13:46.548027 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:13:46.548511 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:13:46.548520 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:13:46.550460 (+ 1940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:13:46.550465 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:13:46.552354 (+ 1889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:13:46.552366 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:13:46.553996 (+ 1630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:13:46.554004 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:13:46.554192 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:13:46.554251 (+ 59us) client_negotiation.cc:769] Sending connection context
0926 15:13:46.554385 (+ 134us) client_negotiation.cc:240] Negotiation successful
0926 15:13:46.554536 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1195,"mutex_wait_us":73,"thread_start_us":535,"threads_started":1}
I20260926 15:13:46.555258 31157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:46.543392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.194:43553 (local address 127.0.0.1:56548)
0926 15:13:46.547568 (+ 4176us) negotiation.cc:107] Waiting for socket to connect
0926 15:13:46.547584 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 15:13:46.547697 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:13:46.548450 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:13:46.548454 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:13:46.548485 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:13:46.548847 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:13:46.548857 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:13:46.550637 (+ 1780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:13:46.550645 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:13:46.552923 (+ 2278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:13:46.552936 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:13:46.554323 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:13:46.554330 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:13:46.554496 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:13:46.554528 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 15:13:46.554618 (+ 90us) client_negotiation.cc:240] Negotiation successful
0926 15:13:46.554740 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3679,"mutex_wait_us":47,"thread_start_us":2704,"threads_started":1}
I20260926 15:13:46.555724 31160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:46.546525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58118 (local address 127.12.34.193:46527)
0926 15:13:46.547165 (+ 640us) server_negotiation.cc:206] Beginning negotiation
0926 15:13:46.547171 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:13:46.547375 (+ 204us) server_negotiation.cc:407] Connection header received
0926 15:13:46.547527 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:13:46.547533 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:13:46.547809 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:13:46.548001 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:13:46.548698 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:46.550306 (+ 1608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:46.552503 (+ 2197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:46.553789 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:46.553930 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:13:46.554158 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0926 15:13:46.555256 (+ 1098us) server_negotiation.cc:299] Negotiation successful
0926 15:13:46.555425 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":490,"thread_start_us":354,"threads_started":1}
I20260926 15:13:46.556278 31155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:46.543155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.195:41203 (local address 127.0.0.1:56240)
0926 15:13:46.544931 (+ 1776us) negotiation.cc:107] Waiting for socket to connect
0926 15:13:46.544982 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 15:13:46.545202 (+ 220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:13:46.547744 (+ 2542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:13:46.547752 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:13:46.547781 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:13:46.548231 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:13:46.548247 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:13:46.550113 (+ 1866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:13:46.550120 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:13:46.552844 (+ 2724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:13:46.552858 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:13:46.555513 (+ 2655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:13:46.555522 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:13:46.555691 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:13:46.555712 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 15:13:46.555770 (+ 58us) client_negotiation.cc:240] Negotiation successful
0926 15:13:46.555927 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1465,"mutex_wait_us":57,"thread_start_us":1290,"threads_started":1}
I20260926 15:13:46.556692 31156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:46.543261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56240 (local address 127.12.34.195:41203)
0926 15:13:46.547088 (+ 3827us) server_negotiation.cc:206] Beginning negotiation
0926 15:13:46.547096 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 15:13:46.547128 (+ 32us) server_negotiation.cc:407] Connection header received
0926 15:13:46.547223 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:13:46.547230 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:13:46.547521 (+ 291us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:13:46.548264 (+ 743us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:13:46.548520 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:46.549788 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:46.553016 (+ 3228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:46.554917 (+ 1901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:46.555044 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:13:46.556131 (+ 1087us) server_negotiation.cc:1035] Waiting for connection context
0926 15:13:46.556198 (+ 67us) server_negotiation.cc:299] Negotiation successful
0926 15:13:46.556386 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3585,"thread_start_us":3463,"threads_started":1}
I20260926 15:13:46.556780 31158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:46.543985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56548 (local address 127.12.34.194:43553)
0926 15:13:46.545890 (+ 1905us) server_negotiation.cc:206] Beginning negotiation
0926 15:13:46.545901 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 15:13:46.547671 (+ 1770us) server_negotiation.cc:407] Connection header received
0926 15:13:46.547855 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:13:46.547860 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:13:46.548069 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:13:46.548251 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:13:46.549010 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:46.550480 (+ 1470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:46.553121 (+ 2641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:46.554095 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:46.554227 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:13:46.554410 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0926 15:13:46.555504 (+ 1094us) server_negotiation.cc:299] Negotiation successful
0926 15:13:46.555668 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1675,"thread_start_us":1547,"threads_started":1}
I20260926 15:13:46.559072 30811 tablet_service.cc:1467] Processing CreateTablet for tablet 79b4dc082f76404792f881c47175047b (DEFAULT_TABLE table=test-table [id=e2cd6e14cfc44a0bbc198c7328df1bd1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:13:46.559865 30945 tablet_service.cc:1467] Processing CreateTablet for tablet 79b4dc082f76404792f881c47175047b (DEFAULT_TABLE table=test-table [id=e2cd6e14cfc44a0bbc198c7328df1bd1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:13:46.559865 31078 tablet_service.cc:1467] Processing CreateTablet for tablet 79b4dc082f76404792f881c47175047b (DEFAULT_TABLE table=test-table [id=e2cd6e14cfc44a0bbc198c7328df1bd1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:13:46.560551 30811 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79b4dc082f76404792f881c47175047b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:13:46.561123 31078 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79b4dc082f76404792f881c47175047b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:13:46.561123 30945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79b4dc082f76404792f881c47175047b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:13:46.577564 31161 tablet_bootstrap.cc:492] T 79b4dc082f76404792f881c47175047b P d8eea60de04541fe8aef7c71f01da2d4: Bootstrap starting.
I20260926 15:13:46.580699 31161 tablet_bootstrap.cc:654] T 79b4dc082f76404792f881c47175047b P d8eea60de04541fe8aef7c71f01da2d4: Neither blocks nor log segments found. Creating new log.
I20260926 15:13:46.580936 31162 tablet_bootstrap.cc:492] T 79b4dc082f76404792f881c47175047b P df798f79f34b4bb7bb0a4bc242fe0ff8: Bootstrap starting.
I20260926 15:13:46.582401 31163 tablet_bootstrap.cc:492] T 79b4dc082f76404792f881c47175047b P 6486c17b21eb4251840e1d8332864e10: Bootstrap starting.
I20260926 15:13:46.582617 31161 log.cc:824] T 79b4dc082f76404792f881c47175047b P d8eea60de04541fe8aef7c71f01da2d4: Log is configured to *not* fsync() on all Append() calls
I20260926 15:13:46.584705 31162 tablet_bootstrap.cc:654] T 79b4dc082f76404792f881c47175047b P df798f79f34b4bb7bb0a4bc242fe0ff8: Neither blocks nor log segments found. Creating new log.
I20260926 15:13:46.585704 31162 log.cc:824] T 79b4dc082f76404792f881c47175047b P df798f79f34b4bb7bb0a4bc242fe0ff8: Log is configured to *not* fsync() on all Append() calls
I20260926 15:13:46.586315 31163 tablet_bootstrap.cc:654] T 79b4dc082f76404792f881c47175047b P 6486c17b21eb4251840e1d8332864e10: Neither blocks nor log segments found. Creating new log.
I20260926 15:13:46.587427 31163 log.cc:824] T 79b4dc082f76404792f881c47175047b P 6486c17b21eb4251840e1d8332864e10: Log is configured to *not* fsync() on all Append() calls
I20260926 15:13:46.588821 31161 tablet_bootstrap.cc:492] T 79b4dc082f76404792f881c47175047b P d8eea60de04541fe8aef7c71f01da2d4: No bootstrap required, opened a new log
I20260926 15:13:46.588835 31162 tablet_bootstrap.cc:492] T 79b4dc082f76404792f881c47175047b P df798f79f34b4bb7bb0a4bc242fe0ff8: No bootstrap required, opened a new log
I20260926 15:13:46.589126 31162 ts_tablet_manager.cc:1397] T 79b4dc082f76404792f881c47175047b P df798f79f34b4bb7bb0a4bc242fe0ff8: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260926 15:13:46.589129 31161 ts_tablet_manager.cc:1397] T 79b4dc082f76404792f881c47175047b P d8eea60de04541fe8aef7c71f01da2d4: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.006s
I20260926 15:13:46.590252 31163 tablet_bootstrap.cc:492] T 79b4dc082f76404792f881c47175047b P 6486c17b21eb4251840e1d8332864e10: No bootstrap required, opened a new log
I20260926 15:13:46.590489 31163 ts_tablet_manager.cc:1397] T 79b4dc082f76404792f881c47175047b P 6486c17b21eb4251840e1d8332864e10: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260926 15:13:46.596421 31161 raft_consensus.cc:348] T 79b4dc082f76404792f881c47175047b P d8eea60de04541fe8aef7c71f01da2d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6486c17b21eb4251840e1d8332864e10" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 41203 } } peers { permanent_uuid: "df798f79f34b4bb7bb0a4bc242fe0ff8" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 43553 } } peers { permanent_uuid: "d8eea60de04541fe8aef7c71f01da2d4" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 46527 } }
I20260926 15:13:46.596760 31161 raft_consensus.cc:374] T 79b4dc082f76404792f881c47175047b P d8eea60de04541fe8aef7c71f01da2d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:13:46.596890 31161 raft_consensus.cc:729] T 79b4dc082f76404792f881c47175047b P d8eea60de04541fe8aef7c71f01da2d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8eea60de04541fe8aef7c71f01da2d4, State: Initialized, Role: FOLLOWER
I20260926 15:13:46.597527 31161 consensus_queue.cc:260] T 79b4dc082f76404792f881c47175047b P d8eea60de04541fe8aef7c71f01da2d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6486c17b21eb4251840e1d8332864e10" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 41203 } } peers { permanent_uuid: "df798f79f34b4bb7bb0a4bc242fe0ff8" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 43553 } } peers { permanent_uuid: "d8eea60de04541fe8aef7c71f01da2d4" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 46527 } }
I20260926 15:13:46.599177 31161 ts_tablet_manager.cc:1428] T 79b4dc082f76404792f881c47175047b P d8eea60de04541fe8aef7c71f01da2d4: Time spent starting tablet: real 0.010s user 0.008s sys 0.001s
I20260926 15:13:46.599378 31162 raft_consensus.cc:348] T 79b4dc082f76404792f881c47175047b P df798f79f34b4bb7bb0a4bc242fe0ff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6486c17b21eb4251840e1d8332864e10" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 41203 } } peers { permanent_uuid: "df798f79f34b4bb7bb0a4bc242fe0ff8" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 43553 } } peers { permanent_uuid: "d8eea60de04541fe8aef7c71f01da2d4" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 46527 } }
I20260926 15:13:46.599768 31162 raft_consensus.cc:374] T 79b4dc082f76404792f881c47175047b P df798f79f34b4bb7bb0a4bc242fe0ff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:13:46.599866 31162 raft_consensus.cc:729] T 79b4dc082f76404792f881c47175047b P df798f79f34b4bb7bb0a4bc242fe0ff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df798f79f34b4bb7bb0a4bc242fe0ff8, State: Initialized, Role: FOLLOWER
I20260926 15:13:46.600493 31162 consensus_queue.cc:260] T 79b4dc082f76404792f881c47175047b P df798f79f34b4bb7bb0a4bc242fe0ff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6486c17b21eb4251840e1d8332864e10" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 41203 } } peers { permanent_uuid: "df798f79f34b4bb7bb0a4bc242fe0ff8" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 43553 } } peers { permanent_uuid: "d8eea60de04541fe8aef7c71f01da2d4" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 46527 } }
I20260926 15:13:46.600692 31163 raft_consensus.cc:348] T 79b4dc082f76404792f881c47175047b P 6486c17b21eb4251840e1d8332864e10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6486c17b21eb4251840e1d8332864e10" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 41203 } } peers { permanent_uuid: "df798f79f34b4bb7bb0a4bc242fe0ff8" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 43553 } } peers { permanent_uuid: "d8eea60de04541fe8aef7c71f01da2d4" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 46527 } }
I20260926 15:13:46.601184 31163 raft_consensus.cc:374] T 79b4dc082f76404792f881c47175047b P 6486c17b21eb4251840e1d8332864e10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:13:46.601308 31163 raft_consensus.cc:729] T 79b4dc082f76404792f881c47175047b P 6486c17b21eb4251840e1d8332864e10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6486c17b21eb4251840e1d8332864e10, State: Initialized, Role: FOLLOWER
I20260926 15:13:46.601999 31010 heartbeater.cc:503] Master 127.12.34.254:45895 was elected leader, sending a full tablet report...
I20260926 15:13:46.601881 31163 consensus_queue.cc:260] T 79b4dc082f76404792f881c47175047b P 6486c17b21eb4251840e1d8332864e10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6486c17b21eb4251840e1d8332864e10" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 41203 } } peers { permanent_uuid: "df798f79f34b4bb7bb0a4bc242fe0ff8" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 43553 } } peers { permanent_uuid: "d8eea60de04541fe8aef7c71f01da2d4" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 46527 } }
I20260926 15:13:46.602294 31162 ts_tablet_manager.cc:1428] T 79b4dc082f76404792f881c47175047b P df798f79f34b4bb7bb0a4bc242fe0ff8: Time spent starting tablet: real 0.013s user 0.007s sys 0.005s
I20260926 15:13:46.602869 31143 heartbeater.cc:503] Master 127.12.34.254:45895 was elected leader, sending a full tablet report...
I20260926 15:13:46.603466 31163 ts_tablet_manager.cc:1428] T 79b4dc082f76404792f881c47175047b P 6486c17b21eb4251840e1d8332864e10: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
W20260926 15:13:46.718845 31144 tablet.cc:2369] T 79b4dc082f76404792f881c47175047b P 6486c17b21eb4251840e1d8332864e10: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 15:13:46.719798 30877 tablet.cc:2369] T 79b4dc082f76404792f881c47175047b P d8eea60de04541fe8aef7c71f01da2d4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 15:13:46.719925 31011 tablet.cc:2369] T 79b4dc082f76404792f881c47175047b P df798f79f34b4bb7bb0a4bc242fe0ff8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 15:13:46.760375 31168 raft_consensus.cc:482] T 79b4dc082f76404792f881c47175047b P df798f79f34b4bb7bb0a4bc242fe0ff8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 15:13:46.760812 31168 raft_consensus.cc:504] T 79b4dc082f76404792f881c47175047b P df798f79f34b4bb7bb0a4bc242fe0ff8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6486c17b21eb4251840e1d8332864e10" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 41203 } } peers { permanent_uuid: "df798f79f34b4bb7bb0a4bc242fe0ff8" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 43553 } } peers { permanent_uuid: "d8eea60de04541fe8aef7c71f01da2d4" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 46527 } }
I20260926 15:13:46.762789 31168 leader_election.cc:288] T 79b4dc082f76404792f881c47175047b P df798f79f34b4bb7bb0a4bc242fe0ff8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6486c17b21eb4251840e1d8332864e10 (127.12.34.195:41203), d8eea60de04541fe8aef7c71f01da2d4 (127.12.34.193:46527)
I20260926 15:13:46.768364 31167 raft_consensus.cc:482] T 79b4dc082f76404792f881c47175047b P d8eea60de04541fe8aef7c71f01da2d4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 15:13:46.768659 31167 raft_consensus.cc:504] T 79b4dc082f76404792f881c47175047b P d8eea60de04541fe8aef7c71f01da2d4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6486c17b21eb4251840e1d8332864e10" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 41203 } } peers { permanent_uuid: "df798f79f34b4bb7bb0a4bc242fe0ff8" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 43553 } } peers { permanent_uuid: "d8eea60de04541fe8aef7c71f01da2d4" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 46527 } }
I20260926 15:13:46.770651 31167 leader_election.cc:288] T 79b4dc082f76404792f881c47175047b P d8eea60de04541fe8aef7c71f01da2d4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6486c17b21eb4251840e1d8332864e10 (127.12.34.195:41203), df798f79f34b4bb7bb0a4bc242fe0ff8 (127.12.34.194:43553)
I20260926 15:13:46.770805 31170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:46.763278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.195:41203 (local address 127.12.34.194:37035)
0926 15:13:46.763944 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0926 15:13:46.763961 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 15:13:46.764160 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:13:46.764654 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:13:46.764657 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:13:46.764682 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:13:46.765123 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:13:46.765136 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:13:46.766461 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:13:46.766466 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:13:46.768219 (+ 1753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:13:46.768234 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:13:46.769855 (+ 1621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:13:46.769861 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:13:46.770034 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:13:46.770059 (+ 25us) client_negotiation.cc:769] Sending connection context
0926 15:13:46.770182 (+ 123us) client_negotiation.cc:240] Negotiation successful
0926 15:13:46.770316 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"mutex_wait_us":44,"thread_start_us":394,"threads_started":1}
I20260926 15:13:46.771315 31156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:46.763357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.194:37035 (local address 127.12.34.195:41203)
0926 15:13:46.763641 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0926 15:13:46.763652 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 15:13:46.764070 (+ 418us) server_negotiation.cc:407] Connection header received
0926 15:13:46.764314 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:13:46.764323 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:13:46.764423 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:13:46.764529 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:13:46.765281 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:46.766328 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:46.768399 (+ 2071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:46.769560 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:46.769826 (+ 266us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:13:46.769909 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 15:13:46.770211 (+ 302us) server_negotiation.cc:299] Negotiation successful
0926 15:13:46.770347 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 15:13:46.771543 31160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:46.764323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.194:44545 (local address 127.12.34.193:46527)
0926 15:13:46.764531 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0926 15:13:46.764536 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:13:46.764863 (+ 327us) server_negotiation.cc:407] Connection header received
0926 15:13:46.765008 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:13:46.765015 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:13:46.765141 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:13:46.765238 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:13:46.765916 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:46.766807 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:46.769101 (+ 2294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:46.770314 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:46.770465 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:13:46.770535 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 15:13:46.771124 (+ 589us) server_negotiation.cc:299] Negotiation successful
0926 15:13:46.771272 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260926 15:13:46.772686 31171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:46.764070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.193:46527 (local address 127.12.34.194:44545)
0926 15:13:46.764746 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0926 15:13:46.764762 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 15:13:46.764877 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:13:46.765285 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:13:46.765289 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:13:46.765317 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:13:46.765720 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:13:46.765733 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:13:46.766988 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:13:46.766992 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:13:46.768909 (+ 1917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:13:46.768924 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:13:46.770702 (+ 1778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:13:46.770708 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:13:46.770915 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:13:46.770963 (+ 48us) client_negotiation.cc:769] Sending connection context
0926 15:13:46.771766 (+ 803us) client_negotiation.cc:240] Negotiation successful
0926 15:13:46.772145 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"mutex_wait_us":1,"thread_start_us":348,"threads_started":1}
I20260926 15:13:46.773501 31098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79b4dc082f76404792f881c47175047b" candidate_uuid: "df798f79f34b4bb7bb0a4bc242fe0ff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6486c17b21eb4251840e1d8332864e10" is_pre_election: true
I20260926 15:13:46.774084 31098 raft_consensus.cc:2454] T 79b4dc082f76404792f881c47175047b P 6486c17b21eb4251840e1d8332864e10 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df798f79f34b4bb7bb0a4bc242fe0ff8 in term 0.
I20260926 15:13:46.774639 30831 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79b4dc082f76404792f881c47175047b" candidate_uuid: "df798f79f34b4bb7bb0a4bc242fe0ff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8eea60de04541fe8aef7c71f01da2d4" is_pre_election: true
I20260926 15:13:46.774883 30901 leader_election.cc:302] T 79b4dc082f76404792f881c47175047b P df798f79f34b4bb7bb0a4bc242fe0ff8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6486c17b21eb4251840e1d8332864e10, df798f79f34b4bb7bb0a4bc242fe0ff8; no voters:
I20260926 15:13:46.775147 30831 raft_consensus.cc:2454] T 79b4dc082f76404792f881c47175047b P d8eea60de04541fe8aef7c71f01da2d4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df798f79f34b4bb7bb0a4bc242fe0ff8 in term 0.
I20260926 15:13:46.775321 31168 raft_consensus.cc:2781] T 79b4dc082f76404792f881c47175047b P df798f79f34b4bb7bb0a4bc242fe0ff8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 15:13:46.775465 31168 raft_consensus.cc:482] T 79b4dc082f76404792f881c47175047b P df798f79f34b4bb7bb0a4bc242fe0ff8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 15:13:46.775571 31168 raft_consensus.cc:3037] T 79b4dc082f76404792f881c47175047b P df798f79f34b4bb7bb0a4bc242fe0ff8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:13:46.778977 31168 raft_consensus.cc:504] T 79b4dc082f76404792f881c47175047b P df798f79f34b4bb7bb0a4bc242fe0ff8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6486c17b21eb4251840e1d8332864e10" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 41203 } } peers { permanent_uuid: "df798f79f34b4bb7bb0a4bc242fe0ff8" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 43553 } } peers { permanent_uuid: "d8eea60de04541fe8aef7c71f01da2d4" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 46527 } }
I20260926 15:13:46.779333 31156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:46.772210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.193:47009 (local address 127.12.34.195:41203)
0926 15:13:46.772378 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0926 15:13:46.772384 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:13:46.772402 (+ 18us) server_negotiation.cc:407] Connection header received
0926 15:13:46.772559 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:13:46.772567 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:13:46.772654 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:13:46.772736 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:13:46.773695 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:46.774961 (+ 1266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:46.777390 (+ 2429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:46.778217 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:46.778424 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:13:46.778504 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 15:13:46.778894 (+ 390us) server_negotiation.cc:299] Negotiation successful
0926 15:13:46.779047 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 15:13:46.779459 31172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:46.770851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.195:41203 (local address 127.12.34.193:47009)
0926 15:13:46.772244 (+ 1393us) negotiation.cc:107] Waiting for socket to connect
0926 15:13:46.772264 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 15:13:46.772388 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:13:46.772943 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:13:46.772947 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:13:46.772984 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:13:46.773522 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:13:46.773538 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:13:46.775129 (+ 1591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:13:46.775134 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:13:46.777231 (+ 2097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:13:46.777244 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:13:46.778456 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:13:46.778463 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:13:46.778678 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:13:46.778738 (+ 60us) client_negotiation.cc:769] Sending connection context
0926 15:13:46.778851 (+ 113us) client_negotiation.cc:240] Negotiation successful
0926 15:13:46.779005 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1247,"mutex_wait_us":53,"thread_start_us":877,"threads_started":1}
I20260926 15:13:46.780184 31168 leader_election.cc:288] T 79b4dc082f76404792f881c47175047b P df798f79f34b4bb7bb0a4bc242fe0ff8 [CANDIDATE]: Term 1 election: Requested vote from peers 6486c17b21eb4251840e1d8332864e10 (127.12.34.195:41203), d8eea60de04541fe8aef7c71f01da2d4 (127.12.34.193:46527)
I20260926 15:13:46.780485 30831 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79b4dc082f76404792f881c47175047b" candidate_uuid: "df798f79f34b4bb7bb0a4bc242fe0ff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8eea60de04541fe8aef7c71f01da2d4"
I20260926 15:13:46.780367 31097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79b4dc082f76404792f881c47175047b" candidate_uuid: "d8eea60de04541fe8aef7c71f01da2d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6486c17b21eb4251840e1d8332864e10" is_pre_election: true
I20260926 15:13:46.780596 31098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79b4dc082f76404792f881c47175047b" candidate_uuid: "df798f79f34b4bb7bb0a4bc242fe0ff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6486c17b21eb4251840e1d8332864e10"
I20260926 15:13:46.780714 30831 raft_consensus.cc:3037] T 79b4dc082f76404792f881c47175047b P d8eea60de04541fe8aef7c71f01da2d4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:13:46.780773 31097 raft_consensus.cc:2454] T 79b4dc082f76404792f881c47175047b P 6486c17b21eb4251840e1d8332864e10 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8eea60de04541fe8aef7c71f01da2d4 in term 0.
I20260926 15:13:46.781625 30767 leader_election.cc:302] T 79b4dc082f76404792f881c47175047b P d8eea60de04541fe8aef7c71f01da2d4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6486c17b21eb4251840e1d8332864e10, d8eea60de04541fe8aef7c71f01da2d4; no voters:
I20260926 15:13:46.782001 31158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:46.771792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.193:53973 (local address 127.12.34.194:43553)
0926 15:13:46.771962 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0926 15:13:46.771968 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:13:46.773644 (+ 1676us) server_negotiation.cc:407] Connection header received
0926 15:13:46.773990 (+ 346us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:13:46.773996 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:13:46.774072 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:13:46.774134 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:13:46.775054 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:46.776310 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:46.778779 (+ 2469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:46.780296 (+ 1517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:46.780451 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:13:46.780545 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 15:13:46.781551 (+ 1006us) server_negotiation.cc:299] Negotiation successful
0926 15:13:46.781716 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 15:13:46.782586 31173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:46.771255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.194:43553 (local address 127.12.34.193:53973)
0926 15:13:46.773521 (+ 2266us) negotiation.cc:107] Waiting for socket to connect
0926 15:13:46.773540 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 15:13:46.773682 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:13:46.774336 (+ 654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:13:46.774340 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:13:46.774373 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:13:46.774824 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:13:46.774837 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:13:46.776455 (+ 1618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:13:46.776460 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:13:46.778613 (+ 2153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:13:46.778630 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:13:46.780733 (+ 2103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:13:46.780741 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:13:46.780952 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:13:46.781388 (+ 436us) client_negotiation.cc:769] Sending connection context
0926 15:13:46.782031 (+ 643us) client_negotiation.cc:240] Negotiation successful
0926 15:13:46.782180 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2109,"mutex_wait_us":1,"thread_start_us":1893,"threads_started":1}
I20260926 15:13:46.783555 30965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79b4dc082f76404792f881c47175047b" candidate_uuid: "d8eea60de04541fe8aef7c71f01da2d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df798f79f34b4bb7bb0a4bc242fe0ff8" is_pre_election: true
I20260926 15:13:46.783824 30831 raft_consensus.cc:2454] T 79b4dc082f76404792f881c47175047b P d8eea60de04541fe8aef7c71f01da2d4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df798f79f34b4bb7bb0a4bc242fe0ff8 in term 1.
I20260926 15:13:46.784060 31167 raft_consensus.cc:2741] T 79b4dc082f76404792f881c47175047b P d8eea60de04541fe8aef7c71f01da2d4 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260926 15:13:46.784130 30965 raft_consensus.cc:2379] T 79b4dc082f76404792f881c47175047b P df798f79f34b4bb7bb0a4bc242fe0ff8 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d8eea60de04541fe8aef7c71f01da2d4 in current term 1: Already voted for candidate df798f79f34b4bb7bb0a4bc242fe0ff8 in this term.
I20260926 15:13:46.784410 30901 leader_election.cc:302] T 79b4dc082f76404792f881c47175047b P df798f79f34b4bb7bb0a4bc242fe0ff8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d8eea60de04541fe8aef7c71f01da2d4, df798f79f34b4bb7bb0a4bc242fe0ff8; no voters: 6486c17b21eb4251840e1d8332864e10
I20260926 15:13:46.784808 31168 raft_consensus.cc:2781] T 79b4dc082f76404792f881c47175047b P df798f79f34b4bb7bb0a4bc242fe0ff8 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:13:46.785393 31168 raft_consensus.cc:686] T 79b4dc082f76404792f881c47175047b P df798f79f34b4bb7bb0a4bc242fe0ff8 [term 1 LEADER]: Becoming Leader. State: Replica: df798f79f34b4bb7bb0a4bc242fe0ff8, State: Running, Role: LEADER
I20260926 15:13:46.785869 31168 consensus_queue.cc:237] T 79b4dc082f76404792f881c47175047b P df798f79f34b4bb7bb0a4bc242fe0ff8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6486c17b21eb4251840e1d8332864e10" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 41203 } } peers { permanent_uuid: "df798f79f34b4bb7bb0a4bc242fe0ff8" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 43553 } } peers { permanent_uuid: "d8eea60de04541fe8aef7c71f01da2d4" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 46527 } }
I20260926 15:13:46.790504 30685 catalog_manager.cc:5495] T 79b4dc082f76404792f881c47175047b P df798f79f34b4bb7bb0a4bc242fe0ff8 reported cstate change: term changed from 0 to 1, leader changed from <none> to df798f79f34b4bb7bb0a4bc242fe0ff8 (127.12.34.194). New cstate: current_term: 1 leader_uuid: "df798f79f34b4bb7bb0a4bc242fe0ff8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6486c17b21eb4251840e1d8332864e10" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 41203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df798f79f34b4bb7bb0a4bc242fe0ff8" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 43553 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8eea60de04541fe8aef7c71f01da2d4" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 46527 } health_report { overall_health: UNKNOWN } } }
I20260926 15:13:46.861460 31158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:46.855629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56554 (local address 127.12.34.194:43553)
0926 15:13:46.855932 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0926 15:13:46.855937 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:13:46.855959 (+ 22us) server_negotiation.cc:407] Connection header received
0926 15:13:46.856105 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:13:46.856109 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:13:46.856202 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:13:46.856340 (+ 138us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 15:13:46.856698 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:46.857609 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:46.858983 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:46.859438 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:46.859554 (+ 116us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:13:46.860195 (+ 641us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 15:13:46.860743 (+ 548us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 15:13:46.860881 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0926 15:13:46.861171 (+ 290us) server_negotiation.cc:299] Negotiation successful
0926 15:13:46.861212 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195}
I20260926 15:13:46.869801 31098 raft_consensus.cc:3037] T 79b4dc082f76404792f881c47175047b P 6486c17b21eb4251840e1d8332864e10 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:13:46.869973 30831 raft_consensus.cc:1261] T 79b4dc082f76404792f881c47175047b P d8eea60de04541fe8aef7c71f01da2d4 [term 1 FOLLOWER]: Refusing update from remote peer df798f79f34b4bb7bb0a4bc242fe0ff8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:13:46.870801 31174 consensus_queue.cc:1035] T 79b4dc082f76404792f881c47175047b P df798f79f34b4bb7bb0a4bc242fe0ff8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8eea60de04541fe8aef7c71f01da2d4" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 46527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:13:46.873232 31098 raft_consensus.cc:1261] T 79b4dc082f76404792f881c47175047b P 6486c17b21eb4251840e1d8332864e10 [term 1 FOLLOWER]: Refusing update from remote peer df798f79f34b4bb7bb0a4bc242fe0ff8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:13:46.873984 31174 consensus_queue.cc:1035] T 79b4dc082f76404792f881c47175047b P df798f79f34b4bb7bb0a4bc242fe0ff8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6486c17b21eb4251840e1d8332864e10" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 41203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:13:46.880708 31177 mvcc.cc:205] Tried to move back new op lower bound from 7333624327648165888 to 7333624327318675456. Current Snapshot: MvccSnapshot[applied={T|T < 7333624327648165888}]
I20260926 15:13:46.896166 30685 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51976:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 15:13:46.896512 30685 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51976:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 15:13:46.901238 30685 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8b45f55caaa84dcfa1ea3e2846646453: Sending DeleteTablet for 3 replicas of tablet 79b4dc082f76404792f881c47175047b
I20260926 15:13:46.902593 31078 tablet_service.cc:1514] Processing DeleteTablet for tablet 79b4dc082f76404792f881c47175047b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:13:46 UTC) from {username='kudu', principal='kudu/127.12.34.254@KRBTEST.COM'} at 127.0.0.1:56240
I20260926 15:13:46.902961 30811 tablet_service.cc:1514] Processing DeleteTablet for tablet 79b4dc082f76404792f881c47175047b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:13:46 UTC) from {username='kudu', principal='kudu/127.12.34.254@KRBTEST.COM'} at 127.0.0.1:58118
I20260926 15:13:46.903385 31187 tablet_replica.cc:330] T 79b4dc082f76404792f881c47175047b P 6486c17b21eb4251840e1d8332864e10: stopping tablet replica
I20260926 15:13:46.903797 31187 raft_consensus.cc:2229] T 79b4dc082f76404792f881c47175047b P 6486c17b21eb4251840e1d8332864e10 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 15:13:46.903831 30945 tablet_service.cc:1514] Processing DeleteTablet for tablet 79b4dc082f76404792f881c47175047b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:13:46 UTC) from {username='kudu', principal='kudu/127.12.34.254@KRBTEST.COM'} at 127.0.0.1:56548
I20260926 15:13:46.904244 31187 raft_consensus.cc:2258] T 79b4dc082f76404792f881c47175047b P 6486c17b21eb4251840e1d8332864e10 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:13:46.906277 12427 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 30748
I20260926 15:13:46.906494 31187 ts_tablet_manager.cc:1905] T 79b4dc082f76404792f881c47175047b P 6486c17b21eb4251840e1d8332864e10: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:13:46.911484 31189 tablet_replica.cc:330] T 79b4dc082f76404792f881c47175047b P df798f79f34b4bb7bb0a4bc242fe0ff8: stopping tablet replica
I20260926 15:13:46.913172 31188 tablet_replica.cc:330] T 79b4dc082f76404792f881c47175047b P d8eea60de04541fe8aef7c71f01da2d4: stopping tablet replica
I20260926 15:13:46.913676 31189 raft_consensus.cc:2229] T 79b4dc082f76404792f881c47175047b P df798f79f34b4bb7bb0a4bc242fe0ff8 [term 1 LEADER]: Raft consensus shutting down.
I20260926 15:13:46.914219 31188 raft_consensus.cc:2229] T 79b4dc082f76404792f881c47175047b P d8eea60de04541fe8aef7c71f01da2d4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 15:13:46.914536 31187 ts_tablet_manager.cc:1918] T 79b4dc082f76404792f881c47175047b P 6486c17b21eb4251840e1d8332864e10: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 15:13:46.914755 31187 log.cc:1196] T 79b4dc082f76404792f881c47175047b P 6486c17b21eb4251840e1d8332864e10: Deleting WAL directory at /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435318278681-12427-0/minicluster-data/ts-2/wal/wals/79b4dc082f76404792f881c47175047b
I20260926 15:13:46.914893 31189 raft_consensus.cc:2258] T 79b4dc082f76404792f881c47175047b P df798f79f34b4bb7bb0a4bc242fe0ff8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:13:46.915446 31188 raft_consensus.cc:2258] T 79b4dc082f76404792f881c47175047b P d8eea60de04541fe8aef7c71f01da2d4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:13:46.915714 31187 ts_tablet_manager.cc:1939] T 79b4dc082f76404792f881c47175047b P 6486c17b21eb4251840e1d8332864e10: Deleting consensus metadata
I20260926 15:13:46.917470 30674 catalog_manager.cc:4841] TS 6486c17b21eb4251840e1d8332864e10 (127.12.34.195:41203): tablet 79b4dc082f76404792f881c47175047b (table test-table [id=e2cd6e14cfc44a0bbc198c7328df1bd1]) successfully deleted
I20260926 15:13:46.919174 31188 ts_tablet_manager.cc:1905] T 79b4dc082f76404792f881c47175047b P d8eea60de04541fe8aef7c71f01da2d4: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 26 15:13:46 dist-test-slave-865v krb5kdc[30642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435624, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.34.193@KRBTEST.COM
I20260926 15:13:46.922309 31189 ts_tablet_manager.cc:1905] T 79b4dc082f76404792f881c47175047b P df798f79f34b4bb7bb0a4bc242fe0ff8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:13:46.925216 31188 ts_tablet_manager.cc:1918] T 79b4dc082f76404792f881c47175047b P d8eea60de04541fe8aef7c71f01da2d4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 15:13:46.925422 31188 log.cc:1196] T 79b4dc082f76404792f881c47175047b P d8eea60de04541fe8aef7c71f01da2d4: Deleting WAL directory at /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435318278681-12427-0/minicluster-data/ts-0/wal/wals/79b4dc082f76404792f881c47175047b
I20260926 15:13:46.926013 31188 ts_tablet_manager.cc:1939] T 79b4dc082f76404792f881c47175047b P d8eea60de04541fe8aef7c71f01da2d4: Deleting consensus metadata
I20260926 15:13:46.927783 30674 catalog_manager.cc:4841] TS d8eea60de04541fe8aef7c71f01da2d4 (127.12.34.193:46527): tablet 79b4dc082f76404792f881c47175047b (table test-table [id=e2cd6e14cfc44a0bbc198c7328df1bd1]) successfully deleted
I20260926 15:13:46.930454 31189 ts_tablet_manager.cc:1918] T 79b4dc082f76404792f881c47175047b P df798f79f34b4bb7bb0a4bc242fe0ff8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 15:13:46.930639 31189 log.cc:1196] T 79b4dc082f76404792f881c47175047b P df798f79f34b4bb7bb0a4bc242fe0ff8: Deleting WAL directory at /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790435318278681-12427-0/minicluster-data/ts-1/wal/wals/79b4dc082f76404792f881c47175047b
I20260926 15:13:46.931356 31189 ts_tablet_manager.cc:1939] T 79b4dc082f76404792f881c47175047b P df798f79f34b4bb7bb0a4bc242fe0ff8: Deleting consensus metadata
I20260926 15:13:46.932123 31160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:46.907335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58134 (local address 127.12.34.193:46527)
0926 15:13:46.907507 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0926 15:13:46.907514 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:13:46.910373 (+ 2859us) server_negotiation.cc:407] Connection header received
0926 15:13:46.910536 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:13:46.910543 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:13:46.910651 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:13:46.910783 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:13:46.912906 (+ 2123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:46.914188 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:46.916298 (+ 2110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:46.917005 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:46.917134 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:13:46.923561 (+ 6427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:13:46.923574 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:13:46.923591 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:13:46.923644 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:13:46.928346 (+ 4702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:13:46.929446 (+ 1100us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:13:46.929455 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:13:46.929460 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:13:46.929610 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:13:46.930082 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:13:46.930089 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:13:46.930092 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:13:46.930580 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:13:46.930936 (+ 356us) server_negotiation.cc:1035] Waiting for connection context
0926 15:13:46.931264 (+ 328us) server_negotiation.cc:299] Negotiation successful
0926 15:13:46.931525 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 15:13:46.932896 30871 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:13:46.933243 30673 catalog_manager.cc:4841] TS df798f79f34b4bb7bb0a4bc242fe0ff8 (127.12.34.194:43553): tablet 79b4dc082f76404792f881c47175047b (table test-table [id=e2cd6e14cfc44a0bbc198c7328df1bd1]) successfully deleted
I20260926 15:13:47.027136 12427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8uoJGv/build/asan/bin/kudu with pid 30748
I20260926 15:13:47.045042 12427 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 30882
Sep 26 15:13:47 dist-test-slave-865v krb5kdc[30642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435624, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.34.194@KRBTEST.COM
I20260926 15:13:47.062036 31158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:47.045965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56560 (local address 127.12.34.194:43553)
0926 15:13:47.046179 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0926 15:13:47.046186 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:13:47.046207 (+ 21us) server_negotiation.cc:407] Connection header received
0926 15:13:47.046310 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:13:47.046316 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:13:47.046419 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:13:47.046533 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:13:47.047942 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:47.049066 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:47.050219 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:47.050909 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:47.051034 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:13:47.055163 (+ 4129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:13:47.055178 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:13:47.055198 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:13:47.055242 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:13:47.059216 (+ 3974us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:13:47.059898 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:13:47.059907 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:13:47.059911 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:13:47.060079 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:13:47.060456 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:13:47.060461 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:13:47.060465 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:13:47.060907 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:13:47.061245 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0926 15:13:47.061450 (+ 205us) server_negotiation.cc:299] Negotiation successful
0926 15:13:47.061665 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 15:13:47.062793 31005 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:13:47.178422 12427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8uoJGv/build/asan/bin/kudu with pid 30882
I20260926 15:13:47.197674 12427 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 31015
Sep 26 15:13:47 dist-test-slave-865v krb5kdc[30642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435624, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.34.195@KRBTEST.COM
I20260926 15:13:47.214090 31156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:47.198446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56242 (local address 127.12.34.195:41203)
0926 15:13:47.198612 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0926 15:13:47.198617 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:13:47.198636 (+ 19us) server_negotiation.cc:407] Connection header received
0926 15:13:47.198772 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:13:47.198777 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:13:47.198851 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:13:47.198937 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:13:47.200308 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:47.201164 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:47.202453 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:47.202852 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:47.202946 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:13:47.207506 (+ 4560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:13:47.207521 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:13:47.207537 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:13:47.207570 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:13:47.210890 (+ 3320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:13:47.211540 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:13:47.211546 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:13:47.211549 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:13:47.211685 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:13:47.212183 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:13:47.212190 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:13:47.212193 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:13:47.212746 (+ 553us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:13:47.213207 (+ 461us) server_negotiation.cc:1035] Waiting for connection context
0926 15:13:47.213538 (+ 331us) server_negotiation.cc:299] Negotiation successful
0926 15:13:47.213716 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 15:13:47.214869 31138 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:13:47.305946 12427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8uoJGv/build/asan/bin/kudu with pid 31015
I20260926 15:13:47.323693 12427 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 30655
I20260926 15:13:47.324779 30717 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:13:47.434755 12427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8uoJGv/build/asan/bin/kudu with pid 30655
2026-09-26T15:13:47Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4335 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-26T15:13:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T15:13:47Z Disabled control of system clock
I20260926 15:13:47.510963 12427 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8uoJGv/build/asan/bin/kudu
/tmp/dist-test-task8uoJGv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790435318278681-12427-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790435318278681-12427-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790435318278681-12427-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.34.254:42571
--webserver_interface=127.12.34.254
--webserver_port=0
--builtin_ntp_servers=127.12.34.212:46229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.34.254:42571
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 15:13:47.831569 31198 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:13:47.831838 31198 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:13:47.831967 31198 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:13:47.839059 31198 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:13:47.839152 31198 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:13:47.839246 31198 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:13:47.839350 31198 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:13:47.839412 31198 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:13:47.848452 31198 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.34.212:46229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790435318278681-12427-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790435318278681-12427-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.34.254:42571
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.34.254:42571
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.34.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790435318278681-12427-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:13:47.849694 31198 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:13:47.851409 31198 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:13:47.862804 31206 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:13:47.862738 31204 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:13:47.863389 31203 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:13:47.863637 31198 server_base.cc:1034] running on GCE node
I20260926 15:13:47.864354 31198 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:13:47.865774 31198 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:13:47.866974 31198 hybrid_clock.cc:648] HybridClock initialized: now 1790435627866929 us; error 32 us; skew 500 ppm
I20260926 15:13:47.867322 31198 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 15:13:47.876619 31198 webserver.cc:460] Webserver started at http://127.12.34.254:34083/ using document root <none> and password file <none>
I20260926 15:13:47.877434 31198 fs_manager.cc:359] Metadata directory not provided
I20260926 15:13:47.877566 31198 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:13:47.877902 31198 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:13:47.880815 31198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790435318278681-12427-0/minicluster-data/master-0/data/instance:
uuid: "26844ad8ea554d3e8debbf752c347c9a"
format_stamp: "Formatted at 2026-09-26 15:13:47 on dist-test-slave-865v"
server_key: "f8c637eae4aa3edda5eb57bc577893c7"
server_key_iv: "0984453212c35162bf8e22d9f551301a"
server_key_version: "encryptionkey@0"
I20260926 15:13:47.881675 31198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790435318278681-12427-0/minicluster-data/master-0/wal/instance:
uuid: "26844ad8ea554d3e8debbf752c347c9a"
format_stamp: "Formatted at 2026-09-26 15:13:47 on dist-test-slave-865v"
server_key: "f8c637eae4aa3edda5eb57bc577893c7"
server_key_iv: "0984453212c35162bf8e22d9f551301a"
server_key_version: "encryptionkey@0"
I20260926 15:13:47.886868 31198 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 15:13:47.890272 31212 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:13:47.891673 31198 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 15:13:47.891922 31198 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790435318278681-12427-0/minicluster-data/master-0/data,/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790435318278681-12427-0/minicluster-data/master-0/wal
uuid: "26844ad8ea554d3e8debbf752c347c9a"
format_stamp: "Formatted at 2026-09-26 15:13:47 on dist-test-slave-865v"
server_key: "f8c637eae4aa3edda5eb57bc577893c7"
server_key_iv: "0984453212c35162bf8e22d9f551301a"
server_key_version: "encryptionkey@0"
I20260926 15:13:47.892140 31198 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790435318278681-12427-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790435318278681-12427-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790435318278681-12427-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:13:47.919993 31198 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:13:47.920892 31198 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:13:47.921240 31198 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:13:47.940297 31198 rpc_server.cc:307] RPC server started. Bound to: 127.12.34.254:42571
I20260926 15:13:47.940341 31263 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.34.254:42571 every 8 connection(s)
I20260926 15:13:47.941716 31198 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
I20260926 15:13:47.945312 31264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:13:47.951865 12427 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8uoJGv/build/asan/bin/kudu as pid 31198
I20260926 15:13:47.952045 12427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790435318278681-12427-0/minicluster-data/master-0/wal/instance
I20260926 15:13:47.952596 12427 external_mini_cluster.cc:1428] Setting key d2ec1dc0ce8014f78fc17d967d52b9ed
I20260926 15:13:47.956071 31264 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26844ad8ea554d3e8debbf752c347c9a: Bootstrap starting.
I20260926 15:13:47.959281 31264 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 26844ad8ea554d3e8debbf752c347c9a: Neither blocks nor log segments found. Creating new log.
I20260926 15:13:47.960294 31264 log.cc:824] T 00000000000000000000000000000000 P 26844ad8ea554d3e8debbf752c347c9a: Log is configured to *not* fsync() on all Append() calls
I20260926 15:13:47.963008 31264 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26844ad8ea554d3e8debbf752c347c9a: No bootstrap required, opened a new log
I20260926 15:13:47.966061 31267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:47.954552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53610 (local address 127.12.34.254:42571)
0926 15:13:47.955481 (+ 929us) server_negotiation.cc:206] Beginning negotiation
0926 15:13:47.955491 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 15:13:47.955529 (+ 38us) server_negotiation.cc:407] Connection header received
0926 15:13:47.957329 (+ 1800us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:13:47.957352 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:13:47.957789 (+ 437us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:13:47.958154 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:13:47.958690 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:47.960057 (+ 1367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:47.961476 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:47.962122 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:47.962324 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:13:47.963496 (+ 1172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:13:47.963504 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:13:47.963519 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:13:47.964041 (+ 522us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:13:47.964131 (+ 90us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:13:47.964151 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:13:47.964276 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0926 15:13:47.964506 (+ 230us) server_negotiation.cc:299] Negotiation successful
0926 15:13:47.964794 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":519,"thread_start_us":317,"threads_started":1}
I20260926 15:13:47.971237 31264 raft_consensus.cc:348] T 00000000000000000000000000000000 P 26844ad8ea554d3e8debbf752c347c9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26844ad8ea554d3e8debbf752c347c9a" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 42571 } }
I20260926 15:13:47.971594 31264 raft_consensus.cc:374] T 00000000000000000000000000000000 P 26844ad8ea554d3e8debbf752c347c9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:13:47.971719 31264 raft_consensus.cc:729] T 00000000000000000000000000000000 P 26844ad8ea554d3e8debbf752c347c9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26844ad8ea554d3e8debbf752c347c9a, State: Initialized, Role: FOLLOWER
I20260926 15:13:47.972297 31264 consensus_queue.cc:260] T 00000000000000000000000000000000 P 26844ad8ea554d3e8debbf752c347c9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26844ad8ea554d3e8debbf752c347c9a" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 42571 } }
I20260926 15:13:47.972536 31264 raft_consensus.cc:388] T 00000000000000000000000000000000 P 26844ad8ea554d3e8debbf752c347c9a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 15:13:47.972688 31264 raft_consensus.cc:482] T 00000000000000000000000000000000 P 26844ad8ea554d3e8debbf752c347c9a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 15:13:47.972867 31264 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 26844ad8ea554d3e8debbf752c347c9a [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:13:47.975297 31264 raft_consensus.cc:504] T 00000000000000000000000000000000 P 26844ad8ea554d3e8debbf752c347c9a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26844ad8ea554d3e8debbf752c347c9a" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 42571 } }
I20260926 15:13:47.975805 31264 leader_election.cc:302] T 00000000000000000000000000000000 P 26844ad8ea554d3e8debbf752c347c9a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 26844ad8ea554d3e8debbf752c347c9a; no voters:
I20260926 15:13:47.976372 31264 leader_election.cc:288] T 00000000000000000000000000000000 P 26844ad8ea554d3e8debbf752c347c9a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 15:13:47.976595 31269 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 26844ad8ea554d3e8debbf752c347c9a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:13:47.977366 31269 raft_consensus.cc:686] T 00000000000000000000000000000000 P 26844ad8ea554d3e8debbf752c347c9a [term 1 LEADER]: Becoming Leader. State: Replica: 26844ad8ea554d3e8debbf752c347c9a, State: Running, Role: LEADER
I20260926 15:13:47.977892 31269 consensus_queue.cc:237] T 00000000000000000000000000000000 P 26844ad8ea554d3e8debbf752c347c9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26844ad8ea554d3e8debbf752c347c9a" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 42571 } }
I20260926 15:13:47.978467 31264 sys_catalog.cc:564] T 00000000000000000000000000000000 P 26844ad8ea554d3e8debbf752c347c9a [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:13:47.981016 31270 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26844ad8ea554d3e8debbf752c347c9a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "26844ad8ea554d3e8debbf752c347c9a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26844ad8ea554d3e8debbf752c347c9a" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 42571 } } }
I20260926 15:13:47.981500 31270 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26844ad8ea554d3e8debbf752c347c9a [sys.catalog]: This master's current role is: LEADER
I20260926 15:13:47.981757 31271 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26844ad8ea554d3e8debbf752c347c9a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 26844ad8ea554d3e8debbf752c347c9a. Latest consensus state: current_term: 1 leader_uuid: "26844ad8ea554d3e8debbf752c347c9a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26844ad8ea554d3e8debbf752c347c9a" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 42571 } } }
I20260926 15:13:47.982054 31271 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26844ad8ea554d3e8debbf752c347c9a [sys.catalog]: This master's current role is: LEADER
I20260926 15:13:47.984553 31278 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:13:47.988903 31278 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 15:13:47.998981 31278 catalog_manager.cc:1347] Generated new cluster ID: d9297a8a496147989e19e67611629d25
I20260926 15:13:47.999189 31278 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 15:13:48.025950 31278 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 15:13:48.027784 31278 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:13:48.038417 31278 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 26844ad8ea554d3e8debbf752c347c9a: Generated new TSK 0
I20260926 15:13:48.039417 31278 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 15:13:48.047394 12427 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8uoJGv/build/asan/bin/kudu
/tmp/dist-test-task8uoJGv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790435318278681-12427-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790435318278681-12427-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790435318278681-12427-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790435318278681-12427-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.34.193:0
--local_ip_for_outbound_sockets=127.12.34.193
--webserver_interface=127.12.34.193
--webserver_port=0
--tserver_master_addrs=127.12.34.254:42571
--builtin_ntp_servers=127.12.34.212:46229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 15:13:48.381247 31288 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:13:48.381621 31288 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:13:48.381784 31288 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:13:48.389051 31288 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:13:48.389194 31288 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:13:48.389374 31288 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.34.193
I20260926 15:13:48.398337 31288 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.34.212:46229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790435318278681-12427-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790435318278681-12427-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.34.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790435318278681-12427-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.34.193
--webserver_port=0
--tserver_master_addrs=127.12.34.254:42571
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.34.193
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790435318278681-12427-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:13:48.399550 31288 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:13:48.401271 31288 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:13:48.414467 31294 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:13:48.414542 31293 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:13:48.414542 31296 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:13:48.418502 31288 server_base.cc:1034] running on GCE node
I20260926 15:13:48.419095 31288 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:13:48.419950 31288 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:13:48.421198 31288 hybrid_clock.cc:648] HybridClock initialized: now 1790435628421164 us; error 43 us; skew 500 ppm
I20260926 15:13:48.421499 31288 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 15:13:48.423681 31288 webserver.cc:460] Webserver started at http://127.12.34.193:41351/ using document root <none> and password file <none>
I20260926 15:13:48.424320 31288 fs_manager.cc:359] Metadata directory not provided
I20260926 15:13:48.424446 31288 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:13:48.424805 31288 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:13:48.427711 31288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790435318278681-12427-0/minicluster-data/ts-0/data/instance:
uuid: "ce29596314c84d3cafbda20ce40f83fd"
format_stamp: "Formatted at 2026-09-26 15:13:48 on dist-test-slave-865v"
server_key: "1bce55d456b7d51ead3aee6b3dc2f536"
server_key_iv: "c99664f27aa088afec625a2ce34df36d"
server_key_version: "encryptionkey@0"
I20260926 15:13:48.428465 31288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790435318278681-12427-0/minicluster-data/ts-0/wal/instance:
uuid: "ce29596314c84d3cafbda20ce40f83fd"
format_stamp: "Formatted at 2026-09-26 15:13:48 on dist-test-slave-865v"
server_key: "1bce55d456b7d51ead3aee6b3dc2f536"
server_key_iv: "c99664f27aa088afec625a2ce34df36d"
server_key_version: "encryptionkey@0"
I20260926 15:13:48.433452 31288 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260926 15:13:48.436821 31302 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:13:48.438158 31288 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 15:13:48.438346 31288 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790435318278681-12427-0/minicluster-data/ts-0/data,/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790435318278681-12427-0/minicluster-data/ts-0/wal
uuid: "ce29596314c84d3cafbda20ce40f83fd"
format_stamp: "Formatted at 2026-09-26 15:13:48 on dist-test-slave-865v"
server_key: "1bce55d456b7d51ead3aee6b3dc2f536"
server_key_iv: "c99664f27aa088afec625a2ce34df36d"
server_key_version: "encryptionkey@0"
I20260926 15:13:48.438565 31288 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790435318278681-12427-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790435318278681-12427-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790435318278681-12427-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:13:48.470172 31288 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:13:48.471026 31288 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:13:48.471344 31288 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:13:48.472440 31288 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:13:48.474076 31288 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:13:48.474205 31288 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:13:48.474340 31288 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:13:48.474427 31288 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:13:48.510474 31288 rpc_server.cc:307] RPC server started. Bound to: 127.12.34.193:36171
I20260926 15:13:48.510576 31414 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.34.193:36171 every 8 connection(s)
I20260926 15:13:48.511899 31288 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790435318278681-12427-0/minicluster-data/ts-0/data/info.pb
I20260926 15:13:48.515079 12427 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8uoJGv/build/asan/bin/kudu as pid 31288
I20260926 15:13:48.515281 12427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790435318278681-12427-0/minicluster-data/ts-0/wal/instance
I20260926 15:13:48.515901 12427 external_mini_cluster.cc:1428] Setting key 31e47ffe7c9dff348710c44117e8df1c
I20260926 15:13:48.518960 12427 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8uoJGv/build/asan/bin/kudu
/tmp/dist-test-task8uoJGv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790435318278681-12427-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790435318278681-12427-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790435318278681-12427-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790435318278681-12427-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.34.194:0
--local_ip_for_outbound_sockets=127.12.34.194
--webserver_interface=127.12.34.194
--webserver_port=0
--tserver_master_addrs=127.12.34.254:42571
--builtin_ntp_servers=127.12.34.212:46229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 15:13:48.526156 31417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:48.514535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.193:54781 (local address 127.12.34.254:42571)
0926 15:13:48.515088 (+ 553us) server_negotiation.cc:206] Beginning negotiation
0926 15:13:48.515096 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 15:13:48.516437 (+ 1341us) server_negotiation.cc:407] Connection header received
0926 15:13:48.518656 (+ 2219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:13:48.518660 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:13:48.518721 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:13:48.518810 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:13:48.520204 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:48.520953 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:48.523352 (+ 2399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:48.523739 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:48.523848 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:13:48.525416 (+ 1568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:13:48.525421 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:13:48.525424 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:13:48.525441 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:13:48.525483 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:13:48.525495 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:13:48.525567 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 15:13:48.525825 (+ 258us) server_negotiation.cc:299] Negotiation successful
0926 15:13:48.525898 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":267,"threads_started":1}
I20260926 15:13:48.527511 31418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:48.515137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:42571 (local address 127.12.34.193:54781)
0926 15:13:48.516253 (+ 1116us) negotiation.cc:107] Waiting for socket to connect
0926 15:13:48.516302 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 15:13:48.518361 (+ 2059us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:13:48.519074 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:13:48.519087 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:13:48.519211 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:13:48.520011 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:13:48.520038 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:13:48.521145 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:13:48.521153 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:13:48.523191 (+ 2038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:13:48.523216 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:13:48.523975 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:13:48.523984 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:13:48.524318 (+ 334us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:13:48.525072 (+ 754us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 15:13:48.525100 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:13:48.525151 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 15:13:48.525246 (+ 95us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:13:48.525594 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:13:48.525606 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:13:48.525617 (+ 11us) client_negotiation.cc:769] Sending connection context
0926 15:13:48.525791 (+ 174us) client_negotiation.cc:240] Negotiation successful
0926 15:13:48.526179 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":559,"thread_start_us":315,"threads_started":1}
I20260926 15:13:48.529045 31415 heartbeater.cc:347] Connected to a master server at 127.12.34.254:42571
I20260926 15:13:48.529400 31415 heartbeater.cc:464] Registering TS with master...
I20260926 15:13:48.530242 31415 heartbeater.cc:511] Master 127.12.34.254:42571 requested a full tablet report, sending...
I20260926 15:13:48.532282 31229 ts_manager.cc:194] Registered new tserver with Master: ce29596314c84d3cafbda20ce40f83fd (127.12.34.193:36171)
I20260926 15:13:48.533784 31229 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.34.193:54781
W20260926 15:13:48.844030 31419 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:13:48.844292 31419 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:13:48.844422 31419 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:13:48.851543 31419 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:13:48.851639 31419 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:13:48.851809 31419 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.34.194
I20260926 15:13:48.860766 31419 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.34.212:46229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790435318278681-12427-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790435318278681-12427-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.34.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790435318278681-12427-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.34.194
--webserver_port=0
--tserver_master_addrs=127.12.34.254:42571
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.34.194
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790435318278681-12427-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:13:48.862035 31419 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:13:48.863746 31419 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:13:48.876719 31424 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:13:48.876689 31425 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:13:48.876756 31427 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:13:48.877453 31419 server_base.cc:1034] running on GCE node
I20260926 15:13:48.878052 31419 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:13:48.878969 31419 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:13:48.880225 31419 hybrid_clock.cc:648] HybridClock initialized: now 1790435628880176 us; error 42 us; skew 500 ppm
I20260926 15:13:48.880514 31419 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 15:13:48.882954 31419 webserver.cc:460] Webserver started at http://127.12.34.194:41451/ using document root <none> and password file <none>
I20260926 15:13:48.883610 31419 fs_manager.cc:359] Metadata directory not provided
I20260926 15:13:48.883692 31419 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:13:48.883945 31419 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:13:48.886718 31419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790435318278681-12427-0/minicluster-data/ts-1/data/instance:
uuid: "a9ac4ade50a34ea894da6837c0a2aa04"
format_stamp: "Formatted at 2026-09-26 15:13:48 on dist-test-slave-865v"
server_key: "ffad617489a1891549657bdac02f3fa4"
server_key_iv: "aa9f69bbcbb90bfadf30255e23099a43"
server_key_version: "encryptionkey@0"
I20260926 15:13:48.887418 31419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790435318278681-12427-0/minicluster-data/ts-1/wal/instance:
uuid: "a9ac4ade50a34ea894da6837c0a2aa04"
format_stamp: "Formatted at 2026-09-26 15:13:48 on dist-test-slave-865v"
server_key: "ffad617489a1891549657bdac02f3fa4"
server_key_iv: "aa9f69bbcbb90bfadf30255e23099a43"
server_key_version: "encryptionkey@0"
I20260926 15:13:48.892318 31419 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 15:13:48.895717 31433 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:13:48.897262 31419 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 15:13:48.897446 31419 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790435318278681-12427-0/minicluster-data/ts-1/data,/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790435318278681-12427-0/minicluster-data/ts-1/wal
uuid: "a9ac4ade50a34ea894da6837c0a2aa04"
format_stamp: "Formatted at 2026-09-26 15:13:48 on dist-test-slave-865v"
server_key: "ffad617489a1891549657bdac02f3fa4"
server_key_iv: "aa9f69bbcbb90bfadf30255e23099a43"
server_key_version: "encryptionkey@0"
I20260926 15:13:48.897658 31419 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790435318278681-12427-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790435318278681-12427-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790435318278681-12427-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:13:48.933302 31419 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:13:48.934175 31419 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:13:48.934487 31419 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:13:48.935473 31419 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:13:48.936942 31419 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:13:48.937067 31419 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:13:48.937256 31419 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:13:48.937345 31419 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:13:48.976366 31419 rpc_server.cc:307] RPC server started. Bound to: 127.12.34.194:41235
I20260926 15:13:48.976418 31545 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.34.194:41235 every 8 connection(s)
I20260926 15:13:48.977685 31419 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790435318278681-12427-0/minicluster-data/ts-1/data/info.pb
I20260926 15:13:48.987186 12427 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8uoJGv/build/asan/bin/kudu as pid 31419
I20260926 15:13:48.987388 12427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790435318278681-12427-0/minicluster-data/ts-1/wal/instance
I20260926 15:13:48.987823 12427 external_mini_cluster.cc:1428] Setting key d5874b5ea38ba33f634f51f0ea05158e
I20260926 15:13:48.990154 31417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:48.980210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.194:48991 (local address 127.12.34.254:42571)
0926 15:13:48.980396 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0926 15:13:48.980400 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:13:48.981869 (+ 1469us) server_negotiation.cc:407] Connection header received
0926 15:13:48.983771 (+ 1902us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:13:48.983777 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:13:48.983873 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:13:48.983986 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:13:48.984972 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:48.985759 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:48.987269 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:48.987801 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:48.987938 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:13:48.989432 (+ 1494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:13:48.989437 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:13:48.989441 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:13:48.989458 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:13:48.989488 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:13:48.989500 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:13:48.989572 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 15:13:48.989847 (+ 275us) server_negotiation.cc:299] Negotiation successful
0926 15:13:48.989939 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 15:13:48.991073 12427 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8uoJGv/build/asan/bin/kudu
/tmp/dist-test-task8uoJGv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790435318278681-12427-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790435318278681-12427-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790435318278681-12427-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790435318278681-12427-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.34.195:0
--local_ip_for_outbound_sockets=127.12.34.195
--webserver_interface=127.12.34.195
--webserver_port=0
--tserver_master_addrs=127.12.34.254:42571
--builtin_ntp_servers=127.12.34.212:46229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 15:13:48.991612 31548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:48.980612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:42571 (local address 127.12.34.194:48991)
0926 15:13:48.981722 (+ 1110us) negotiation.cc:107] Waiting for socket to connect
0926 15:13:48.981766 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 15:13:48.983483 (+ 1717us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:13:48.984199 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:13:48.984206 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:13:48.984285 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:13:48.984799 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:13:48.984813 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:13:48.985913 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:13:48.985924 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:13:48.987124 (+ 1200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:13:48.987136 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:13:48.988033 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:13:48.988041 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:13:48.988255 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:13:48.989068 (+ 813us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 15:13:48.989071 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:13:48.989146 (+ 75us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 15:13:48.989241 (+ 95us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:13:48.989604 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:13:48.989616 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:13:48.989624 (+ 8us) client_negotiation.cc:769] Sending connection context
0926 15:13:48.989837 (+ 213us) client_negotiation.cc:240] Negotiation successful
0926 15:13:48.990198 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":621,"thread_start_us":278,"threads_started":1}
I20260926 15:13:48.992952 31546 heartbeater.cc:347] Connected to a master server at 127.12.34.254:42571
I20260926 15:13:48.993297 31546 heartbeater.cc:464] Registering TS with master...
I20260926 15:13:48.994014 31546 heartbeater.cc:511] Master 127.12.34.254:42571 requested a full tablet report, sending...
I20260926 15:13:48.995329 31229 ts_manager.cc:194] Registered new tserver with Master: a9ac4ade50a34ea894da6837c0a2aa04 (127.12.34.194:41235)
I20260926 15:13:48.996233 31229 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.34.194:48991
W20260926 15:13:49.311100 31549 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:13:49.311421 31549 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:13:49.311556 31549 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:13:49.318812 31549 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:13:49.318907 31549 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:13:49.319060 31549 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.34.195
I20260926 15:13:49.328042 31549 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.34.212:46229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790435318278681-12427-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790435318278681-12427-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.34.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790435318278681-12427-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.34.195
--webserver_port=0
--tserver_master_addrs=127.12.34.254:42571
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.34.195
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790435318278681-12427-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:13:49.329324 31549 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:13:49.331027 31549 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:13:49.345582 31554 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:13:49.345623 31557 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:13:49.345665 31555 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:13:49.346333 31549 server_base.cc:1034] running on GCE node
I20260926 15:13:49.346876 31549 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:13:49.347735 31549 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:13:49.348976 31549 hybrid_clock.cc:648] HybridClock initialized: now 1790435629348939 us; error 54 us; skew 500 ppm
I20260926 15:13:49.349318 31549 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 15:13:49.351544 31549 webserver.cc:460] Webserver started at http://127.12.34.195:44159/ using document root <none> and password file <none>
I20260926 15:13:49.352201 31549 fs_manager.cc:359] Metadata directory not provided
I20260926 15:13:49.352334 31549 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:13:49.352646 31549 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:13:49.355515 31549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790435318278681-12427-0/minicluster-data/ts-2/data/instance:
uuid: "180820963b6845c1bfe7e9ecde750a3d"
format_stamp: "Formatted at 2026-09-26 15:13:49 on dist-test-slave-865v"
server_key: "772dc375369b0ae59bcc7f021231ba74"
server_key_iv: "eba540ed31821bcd1e141e8cb575e900"
server_key_version: "encryptionkey@0"
I20260926 15:13:49.356287 31549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790435318278681-12427-0/minicluster-data/ts-2/wal/instance:
uuid: "180820963b6845c1bfe7e9ecde750a3d"
format_stamp: "Formatted at 2026-09-26 15:13:49 on dist-test-slave-865v"
server_key: "772dc375369b0ae59bcc7f021231ba74"
server_key_iv: "eba540ed31821bcd1e141e8cb575e900"
server_key_version: "encryptionkey@0"
I20260926 15:13:49.360988 31549 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260926 15:13:49.364270 31563 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:13:49.365561 31549 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 15:13:49.365733 31549 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790435318278681-12427-0/minicluster-data/ts-2/data,/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790435318278681-12427-0/minicluster-data/ts-2/wal
uuid: "180820963b6845c1bfe7e9ecde750a3d"
format_stamp: "Formatted at 2026-09-26 15:13:49 on dist-test-slave-865v"
server_key: "772dc375369b0ae59bcc7f021231ba74"
server_key_iv: "eba540ed31821bcd1e141e8cb575e900"
server_key_version: "encryptionkey@0"
I20260926 15:13:49.365939 31549 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790435318278681-12427-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790435318278681-12427-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790435318278681-12427-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:13:49.396322 31549 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:13:49.397457 31549 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:13:49.397763 31549 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:13:49.398746 31549 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:13:49.400208 31549 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:13:49.400352 31549 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:13:49.400477 31549 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:13:49.400563 31549 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:13:49.434479 31549 rpc_server.cc:307] RPC server started. Bound to: 127.12.34.195:46673
I20260926 15:13:49.434566 31675 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.34.195:46673 every 8 connection(s)
I20260926 15:13:49.435813 31549 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790435318278681-12427-0/minicluster-data/ts-2/data/info.pb
I20260926 15:13:49.445839 12427 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8uoJGv/build/asan/bin/kudu as pid 31549
I20260926 15:13:49.446018 12427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790435318278681-12427-0/minicluster-data/ts-2/wal/instance
I20260926 15:13:49.446446 12427 external_mini_cluster.cc:1428] Setting key 5d07e95f1cb120cfb1e65528381b905e
I20260926 15:13:49.447681 31417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:49.438324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.195:56903 (local address 127.12.34.254:42571)
0926 15:13:49.438513 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0926 15:13:49.438517 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:13:49.439742 (+ 1225us) server_negotiation.cc:407] Connection header received
0926 15:13:49.441498 (+ 1756us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:13:49.441503 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:13:49.441581 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:13:49.441678 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:13:49.442739 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:49.443521 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:49.444967 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:49.445408 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:49.445554 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:13:49.446875 (+ 1321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:13:49.446886 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:13:49.446890 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:13:49.446908 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:13:49.446938 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:13:49.446950 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:13:49.447021 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 15:13:49.447288 (+ 267us) server_negotiation.cc:299] Negotiation successful
0926 15:13:49.447378 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 15:13:49.448844 31678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:49.438779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:42571 (local address 127.12.34.195:56903)
0926 15:13:49.439608 (+ 829us) negotiation.cc:107] Waiting for socket to connect
0926 15:13:49.439641 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0926 15:13:49.441236 (+ 1595us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:13:49.441873 (+ 637us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:13:49.441880 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:13:49.441960 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:13:49.442531 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:13:49.442557 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:13:49.443643 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:13:49.443647 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:13:49.444841 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:13:49.444854 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:13:49.445616 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:13:49.445621 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:13:49.445794 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:13:49.446541 (+ 747us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 15:13:49.446546 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:13:49.446603 (+ 57us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 15:13:49.446683 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:13:49.447057 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:13:49.447069 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:13:49.447077 (+ 8us) client_negotiation.cc:769] Sending connection context
0926 15:13:49.447258 (+ 181us) client_negotiation.cc:240] Negotiation successful
0926 15:13:49.447595 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":251,"threads_started":1}
I20260926 15:13:49.450073 31676 heartbeater.cc:347] Connected to a master server at 127.12.34.254:42571
I20260926 15:13:49.450307 31676 heartbeater.cc:464] Registering TS with master...
I20260926 15:13:49.450934 31676 heartbeater.cc:511] Master 127.12.34.254:42571 requested a full tablet report, sending...
I20260926 15:13:49.452127 31229 ts_manager.cc:194] Registered new tserver with Master: 180820963b6845c1bfe7e9ecde750a3d (127.12.34.195:46673)
I20260926 15:13:49.452847 31229 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.34.195:56903
I20260926 15:13:49.461359 12427 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 15:13:49.470726 31417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:49.465180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46530 (local address 127.12.34.254:42571)
0926 15:13:49.465403 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0926 15:13:49.465412 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 15:13:49.465758 (+ 346us) server_negotiation.cc:407] Connection header received
0926 15:13:49.465883 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:13:49.465887 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:13:49.465952 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:13:49.466028 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:13:49.466412 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:49.467084 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:49.468318 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:49.468799 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:49.468889 (+ 90us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:13:49.470071 (+ 1182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:13:49.470076 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:13:49.470079 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:13:49.470094 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:13:49.470122 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:13:49.470133 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:13:49.470261 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0926 15:13:49.470369 (+ 108us) server_negotiation.cc:299] Negotiation successful
0926 15:13:49.470425 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 15:13:49.474431 31229 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46530:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 15:13:49.481467 31229 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 15:13:49.515806 31691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:49.493873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43892 (local address 127.12.34.193:36171)
0926 15:13:49.502560 (+ 8687us) server_negotiation.cc:206] Beginning negotiation
0926 15:13:49.502573 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 15:13:49.504146 (+ 1573us) server_negotiation.cc:407] Connection header received
0926 15:13:49.504547 (+ 401us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:13:49.504558 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:13:49.504811 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:13:49.504992 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:13:49.506222 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:49.508426 (+ 2204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:49.512130 (+ 3704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:49.514074 (+ 1944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:49.514180 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:13:49.514487 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0926 15:13:49.515156 (+ 669us) server_negotiation.cc:299] Negotiation successful
0926 15:13:49.515355 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8453,"thread_start_us":8294,"threads_started":1}
I20260926 15:13:49.516840 31690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:49.493191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.193:36171 (local address 127.0.0.1:43892)
0926 15:13:49.504010 (+ 10819us) negotiation.cc:107] Waiting for socket to connect
0926 15:13:49.504031 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 15:13:49.504339 (+ 308us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:13:49.505369 (+ 1030us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:13:49.505377 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:13:49.505411 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:13:49.505977 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:13:49.505992 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:13:49.508671 (+ 2679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:13:49.508681 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:13:49.511854 (+ 3173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:13:49.511876 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:13:49.514647 (+ 2771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:13:49.514654 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:13:49.514881 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:13:49.514956 (+ 75us) client_negotiation.cc:769] Sending connection context
0926 15:13:49.516348 (+ 1392us) client_negotiation.cc:240] Negotiation successful
0926 15:13:49.516543 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9895,"thread_start_us":9539,"threads_started":1}
I20260926 15:13:49.519224 31688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:49.493194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.194:41235 (local address 127.0.0.1:60220)
0926 15:13:49.503391 (+ 10197us) negotiation.cc:107] Waiting for socket to connect
0926 15:13:49.503430 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 15:13:49.503626 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:13:49.504623 (+ 997us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:13:49.504633 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:13:49.504685 (+ 52us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:13:49.506158 (+ 1473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:13:49.506177 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:13:49.508885 (+ 2708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:13:49.508894 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:13:49.516545 (+ 7651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:13:49.516560 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:13:49.518430 (+ 1870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:13:49.518437 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:13:49.518615 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:13:49.518656 (+ 41us) client_negotiation.cc:769] Sending connection context
0926 15:13:49.518755 (+ 99us) client_negotiation.cc:240] Negotiation successful
0926 15:13:49.518879 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9929,"mutex_wait_us":249,"thread_start_us":8475,"threads_started":1}
I20260926 15:13:49.519349 31689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:49.493290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60220 (local address 127.12.34.194:41235)
0926 15:13:49.503272 (+ 9982us) server_negotiation.cc:206] Beginning negotiation
0926 15:13:49.503277 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:13:49.503555 (+ 278us) server_negotiation.cc:407] Connection header received
0926 15:13:49.503898 (+ 343us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:13:49.503910 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:13:49.504225 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:13:49.506413 (+ 2188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:13:49.506714 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:49.508681 (+ 1967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:49.516805 (+ 8124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:49.518224 (+ 1419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:49.518373 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:13:49.518574 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0926 15:13:49.518836 (+ 262us) server_negotiation.cc:299] Negotiation successful
0926 15:13:49.519009 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9835,"thread_start_us":9745,"threads_started":1}
I20260926 15:13:49.520995 31350 tablet_service.cc:1467] Processing CreateTablet for tablet 0a0d0666b51b4c24b213cbe30bb92e85 (DEFAULT_TABLE table=test-table [id=11e3ca4a4ad84b41820873ad9525b1c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:13:49.522505 31350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a0d0666b51b4c24b213cbe30bb92e85. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:13:49.522521 31481 tablet_service.cc:1467] Processing CreateTablet for tablet 0a0d0666b51b4c24b213cbe30bb92e85 (DEFAULT_TABLE table=test-table [id=11e3ca4a4ad84b41820873ad9525b1c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:13:49.523229 31692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:49.503467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.195:46673 (local address 127.0.0.1:35534)
0926 15:13:49.513226 (+ 9759us) negotiation.cc:107] Waiting for socket to connect
0926 15:13:49.513245 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 15:13:49.514210 (+ 965us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:13:49.515098 (+ 888us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:13:49.515104 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:13:49.515129 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:13:49.515568 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:13:49.515580 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:13:49.517520 (+ 1940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:13:49.517525 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:13:49.520608 (+ 3083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:13:49.520624 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:13:49.522508 (+ 1884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:13:49.522516 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:13:49.522687 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:13:49.522720 (+ 33us) client_negotiation.cc:769] Sending connection context
0926 15:13:49.522827 (+ 107us) client_negotiation.cc:240] Negotiation successful
0926 15:13:49.522949 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9636,"thread_start_us":10428,"threads_started":1}
I20260926 15:13:49.523398 31693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:49.503576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35534 (local address 127.12.34.195:46673)
0926 15:13:49.513829 (+ 10253us) server_negotiation.cc:206] Beginning negotiation
0926 15:13:49.513848 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 15:13:49.513882 (+ 34us) server_negotiation.cc:407] Connection header received
0926 15:13:49.514401 (+ 519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:13:49.514408 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:13:49.514649 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:13:49.514827 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:13:49.515799 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:49.517359 (+ 1560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:49.520891 (+ 3532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:49.522291 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:49.522437 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:13:49.522701 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0926 15:13:49.522878 (+ 177us) server_negotiation.cc:299] Negotiation successful
0926 15:13:49.523033 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10082,"thread_start_us":9930,"threads_started":1}
I20260926 15:13:49.523799 31481 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a0d0666b51b4c24b213cbe30bb92e85. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:13:49.526854 31611 tablet_service.cc:1467] Processing CreateTablet for tablet 0a0d0666b51b4c24b213cbe30bb92e85 (DEFAULT_TABLE table=test-table [id=11e3ca4a4ad84b41820873ad9525b1c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:13:49.528128 31611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a0d0666b51b4c24b213cbe30bb92e85. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:13:49.534420 31694 tablet_bootstrap.cc:492] T 0a0d0666b51b4c24b213cbe30bb92e85 P a9ac4ade50a34ea894da6837c0a2aa04: Bootstrap starting.
I20260926 15:13:49.537230 31415 heartbeater.cc:503] Master 127.12.34.254:42571 was elected leader, sending a full tablet report...
I20260926 15:13:49.537846 31694 tablet_bootstrap.cc:654] T 0a0d0666b51b4c24b213cbe30bb92e85 P a9ac4ade50a34ea894da6837c0a2aa04: Neither blocks nor log segments found. Creating new log.
I20260926 15:13:49.539136 31694 log.cc:824] T 0a0d0666b51b4c24b213cbe30bb92e85 P a9ac4ade50a34ea894da6837c0a2aa04: Log is configured to *not* fsync() on all Append() calls
I20260926 15:13:49.541949 31695 tablet_bootstrap.cc:492] T 0a0d0666b51b4c24b213cbe30bb92e85 P ce29596314c84d3cafbda20ce40f83fd: Bootstrap starting.
I20260926 15:13:49.543434 31697 tablet_bootstrap.cc:492] T 0a0d0666b51b4c24b213cbe30bb92e85 P 180820963b6845c1bfe7e9ecde750a3d: Bootstrap starting.
I20260926 15:13:49.545915 31695 tablet_bootstrap.cc:654] T 0a0d0666b51b4c24b213cbe30bb92e85 P ce29596314c84d3cafbda20ce40f83fd: Neither blocks nor log segments found. Creating new log.
I20260926 15:13:49.546411 31697 tablet_bootstrap.cc:654] T 0a0d0666b51b4c24b213cbe30bb92e85 P 180820963b6845c1bfe7e9ecde750a3d: Neither blocks nor log segments found. Creating new log.
I20260926 15:13:49.547125 31695 log.cc:824] T 0a0d0666b51b4c24b213cbe30bb92e85 P ce29596314c84d3cafbda20ce40f83fd: Log is configured to *not* fsync() on all Append() calls
I20260926 15:13:49.547360 31697 log.cc:824] T 0a0d0666b51b4c24b213cbe30bb92e85 P 180820963b6845c1bfe7e9ecde750a3d: Log is configured to *not* fsync() on all Append() calls
I20260926 15:13:49.547822 31694 tablet_bootstrap.cc:492] T 0a0d0666b51b4c24b213cbe30bb92e85 P a9ac4ade50a34ea894da6837c0a2aa04: No bootstrap required, opened a new log
I20260926 15:13:49.548143 31694 ts_tablet_manager.cc:1397] T 0a0d0666b51b4c24b213cbe30bb92e85 P a9ac4ade50a34ea894da6837c0a2aa04: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.004s
I20260926 15:13:49.550190 31695 tablet_bootstrap.cc:492] T 0a0d0666b51b4c24b213cbe30bb92e85 P ce29596314c84d3cafbda20ce40f83fd: No bootstrap required, opened a new log
I20260926 15:13:49.550462 31695 ts_tablet_manager.cc:1397] T 0a0d0666b51b4c24b213cbe30bb92e85 P ce29596314c84d3cafbda20ce40f83fd: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260926 15:13:49.554921 31697 tablet_bootstrap.cc:492] T 0a0d0666b51b4c24b213cbe30bb92e85 P 180820963b6845c1bfe7e9ecde750a3d: No bootstrap required, opened a new log
I20260926 15:13:49.555167 31697 ts_tablet_manager.cc:1397] T 0a0d0666b51b4c24b213cbe30bb92e85 P 180820963b6845c1bfe7e9ecde750a3d: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.003s
I20260926 15:13:49.558562 31694 raft_consensus.cc:348] T 0a0d0666b51b4c24b213cbe30bb92e85 P a9ac4ade50a34ea894da6837c0a2aa04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9ac4ade50a34ea894da6837c0a2aa04" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 41235 } } peers { permanent_uuid: "ce29596314c84d3cafbda20ce40f83fd" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 36171 } } peers { permanent_uuid: "180820963b6845c1bfe7e9ecde750a3d" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 46673 } }
I20260926 15:13:49.559005 31694 raft_consensus.cc:374] T 0a0d0666b51b4c24b213cbe30bb92e85 P a9ac4ade50a34ea894da6837c0a2aa04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:13:49.559136 31694 raft_consensus.cc:729] T 0a0d0666b51b4c24b213cbe30bb92e85 P a9ac4ade50a34ea894da6837c0a2aa04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9ac4ade50a34ea894da6837c0a2aa04, State: Initialized, Role: FOLLOWER
I20260926 15:13:49.559727 31694 consensus_queue.cc:260] T 0a0d0666b51b4c24b213cbe30bb92e85 P a9ac4ade50a34ea894da6837c0a2aa04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9ac4ade50a34ea894da6837c0a2aa04" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 41235 } } peers { permanent_uuid: "ce29596314c84d3cafbda20ce40f83fd" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 36171 } } peers { permanent_uuid: "180820963b6845c1bfe7e9ecde750a3d" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 46673 } }
I20260926 15:13:49.560626 31695 raft_consensus.cc:348] T 0a0d0666b51b4c24b213cbe30bb92e85 P ce29596314c84d3cafbda20ce40f83fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9ac4ade50a34ea894da6837c0a2aa04" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 41235 } } peers { permanent_uuid: "ce29596314c84d3cafbda20ce40f83fd" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 36171 } } peers { permanent_uuid: "180820963b6845c1bfe7e9ecde750a3d" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 46673 } }
I20260926 15:13:49.560894 31546 heartbeater.cc:503] Master 127.12.34.254:42571 was elected leader, sending a full tablet report...
I20260926 15:13:49.561010 31695 raft_consensus.cc:374] T 0a0d0666b51b4c24b213cbe30bb92e85 P ce29596314c84d3cafbda20ce40f83fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:13:49.561184 31695 raft_consensus.cc:729] T 0a0d0666b51b4c24b213cbe30bb92e85 P ce29596314c84d3cafbda20ce40f83fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce29596314c84d3cafbda20ce40f83fd, State: Initialized, Role: FOLLOWER
I20260926 15:13:49.561653 31694 ts_tablet_manager.cc:1428] T 0a0d0666b51b4c24b213cbe30bb92e85 P a9ac4ade50a34ea894da6837c0a2aa04: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260926 15:13:49.561782 31695 consensus_queue.cc:260] T 0a0d0666b51b4c24b213cbe30bb92e85 P ce29596314c84d3cafbda20ce40f83fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9ac4ade50a34ea894da6837c0a2aa04" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 41235 } } peers { permanent_uuid: "ce29596314c84d3cafbda20ce40f83fd" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 36171 } } peers { permanent_uuid: "180820963b6845c1bfe7e9ecde750a3d" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 46673 } }
I20260926 15:13:49.564868 31695 ts_tablet_manager.cc:1428] T 0a0d0666b51b4c24b213cbe30bb92e85 P ce29596314c84d3cafbda20ce40f83fd: Time spent starting tablet: real 0.014s user 0.009s sys 0.004s
I20260926 15:13:49.564637 31697 raft_consensus.cc:348] T 0a0d0666b51b4c24b213cbe30bb92e85 P 180820963b6845c1bfe7e9ecde750a3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9ac4ade50a34ea894da6837c0a2aa04" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 41235 } } peers { permanent_uuid: "ce29596314c84d3cafbda20ce40f83fd" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 36171 } } peers { permanent_uuid: "180820963b6845c1bfe7e9ecde750a3d" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 46673 } }
I20260926 15:13:49.565063 31697 raft_consensus.cc:374] T 0a0d0666b51b4c24b213cbe30bb92e85 P 180820963b6845c1bfe7e9ecde750a3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:13:49.565224 31697 raft_consensus.cc:729] T 0a0d0666b51b4c24b213cbe30bb92e85 P 180820963b6845c1bfe7e9ecde750a3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 180820963b6845c1bfe7e9ecde750a3d, State: Initialized, Role: FOLLOWER
I20260926 15:13:49.565769 31697 consensus_queue.cc:260] T 0a0d0666b51b4c24b213cbe30bb92e85 P 180820963b6845c1bfe7e9ecde750a3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9ac4ade50a34ea894da6837c0a2aa04" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 41235 } } peers { permanent_uuid: "ce29596314c84d3cafbda20ce40f83fd" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 36171 } } peers { permanent_uuid: "180820963b6845c1bfe7e9ecde750a3d" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 46673 } }
I20260926 15:13:49.567044 31676 heartbeater.cc:503] Master 127.12.34.254:42571 was elected leader, sending a full tablet report...
I20260926 15:13:49.567320 31697 ts_tablet_manager.cc:1428] T 0a0d0666b51b4c24b213cbe30bb92e85 P 180820963b6845c1bfe7e9ecde750a3d: Time spent starting tablet: real 0.012s user 0.006s sys 0.005s
W20260926 15:13:49.687525 31677 tablet.cc:2369] T 0a0d0666b51b4c24b213cbe30bb92e85 P 180820963b6845c1bfe7e9ecde750a3d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 15:13:49.729900 31547 tablet.cc:2369] T 0a0d0666b51b4c24b213cbe30bb92e85 P a9ac4ade50a34ea894da6837c0a2aa04: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 15:13:49.733572 31700 raft_consensus.cc:482] T 0a0d0666b51b4c24b213cbe30bb92e85 P a9ac4ade50a34ea894da6837c0a2aa04 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 15:13:49.733871 31700 raft_consensus.cc:504] T 0a0d0666b51b4c24b213cbe30bb92e85 P a9ac4ade50a34ea894da6837c0a2aa04 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9ac4ade50a34ea894da6837c0a2aa04" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 41235 } } peers { permanent_uuid: "ce29596314c84d3cafbda20ce40f83fd" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 36171 } } peers { permanent_uuid: "180820963b6845c1bfe7e9ecde750a3d" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 46673 } }
I20260926 15:13:49.736415 31700 leader_election.cc:288] T 0a0d0666b51b4c24b213cbe30bb92e85 P a9ac4ade50a34ea894da6837c0a2aa04 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ce29596314c84d3cafbda20ce40f83fd (127.12.34.193:36171), 180820963b6845c1bfe7e9ecde750a3d (127.12.34.195:46673)
I20260926 15:13:49.737912 31702 raft_consensus.cc:482] T 0a0d0666b51b4c24b213cbe30bb92e85 P 180820963b6845c1bfe7e9ecde750a3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 15:13:49.738241 31702 raft_consensus.cc:504] T 0a0d0666b51b4c24b213cbe30bb92e85 P 180820963b6845c1bfe7e9ecde750a3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9ac4ade50a34ea894da6837c0a2aa04" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 41235 } } peers { permanent_uuid: "ce29596314c84d3cafbda20ce40f83fd" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 36171 } } peers { permanent_uuid: "180820963b6845c1bfe7e9ecde750a3d" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 46673 } }
I20260926 15:13:49.740118 31702 leader_election.cc:288] T 0a0d0666b51b4c24b213cbe30bb92e85 P 180820963b6845c1bfe7e9ecde750a3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a9ac4ade50a34ea894da6837c0a2aa04 (127.12.34.194:41235), ce29596314c84d3cafbda20ce40f83fd (127.12.34.193:36171)
I20260926 15:13:49.743948 31703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:49.735863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.195:46673 (local address 127.12.34.194:37081)
0926 15:13:49.736527 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0926 15:13:49.736549 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 15:13:49.736688 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:13:49.737126 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:13:49.737133 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:13:49.737166 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:13:49.737545 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:13:49.737562 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:13:49.739083 (+ 1521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:13:49.739091 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:13:49.741306 (+ 2215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:13:49.741320 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:13:49.743158 (+ 1838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:13:49.743167 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:13:49.743372 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:13:49.743410 (+ 38us) client_negotiation.cc:769] Sending connection context
0926 15:13:49.743492 (+ 82us) client_negotiation.cc:240] Negotiation successful
0926 15:13:49.743625 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":310,"threads_started":1}
I20260926 15:13:49.743956 31691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:49.736451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.194:57293 (local address 127.12.34.193:36171)
0926 15:13:49.736655 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0926 15:13:49.736662 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:13:49.737153 (+ 491us) server_negotiation.cc:407] Connection header received
0926 15:13:49.737382 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:13:49.737387 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:13:49.737481 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:13:49.737583 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:13:49.738259 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:49.739304 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:49.741682 (+ 2378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:49.742914 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:49.743034 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:13:49.743103 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 15:13:49.743604 (+ 501us) server_negotiation.cc:299] Negotiation successful
0926 15:13:49.743737 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 15:13:49.744750 31704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:49.735952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.193:36171 (local address 127.12.34.194:57293)
0926 15:13:49.737053 (+ 1101us) negotiation.cc:107] Waiting for socket to connect
0926 15:13:49.737070 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 15:13:49.737232 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:13:49.737621 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:13:49.737625 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:13:49.737649 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:13:49.738023 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:13:49.738035 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:13:49.739457 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:13:49.739462 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:13:49.741474 (+ 2012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:13:49.741489 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:13:49.743282 (+ 1793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:13:49.743288 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:13:49.743432 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:13:49.743473 (+ 41us) client_negotiation.cc:769] Sending connection context
0926 15:13:49.744259 (+ 786us) client_negotiation.cc:240] Negotiation successful
0926 15:13:49.744393 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":965,"thread_start_us":795,"threads_started":1}
I20260926 15:13:49.745373 31693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:49.736456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.194:37081 (local address 127.12.34.195:46673)
0926 15:13:49.736669 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0926 15:13:49.736679 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 15:13:49.736706 (+ 27us) server_negotiation.cc:407] Connection header received
0926 15:13:49.736867 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:13:49.736874 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:13:49.736958 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:13:49.737059 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:13:49.737861 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:49.738957 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:49.741533 (+ 2576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:49.742786 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:49.742925 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:13:49.744842 (+ 1917us) server_negotiation.cc:1035] Waiting for connection context
0926 15:13:49.744896 (+ 54us) server_negotiation.cc:299] Negotiation successful
0926 15:13:49.745016 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1806,"server-negotiator.queue_time_us":72}
I20260926 15:13:49.746084 31370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a0d0666b51b4c24b213cbe30bb92e85" candidate_uuid: "a9ac4ade50a34ea894da6837c0a2aa04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce29596314c84d3cafbda20ce40f83fd" is_pre_election: true
I20260926 15:13:49.746088 31631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a0d0666b51b4c24b213cbe30bb92e85" candidate_uuid: "a9ac4ade50a34ea894da6837c0a2aa04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "180820963b6845c1bfe7e9ecde750a3d" is_pre_election: true
I20260926 15:13:49.746593 31631 raft_consensus.cc:2454] T 0a0d0666b51b4c24b213cbe30bb92e85 P 180820963b6845c1bfe7e9ecde750a3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a9ac4ade50a34ea894da6837c0a2aa04 in term 0.
I20260926 15:13:49.746651 31370 raft_consensus.cc:2454] T 0a0d0666b51b4c24b213cbe30bb92e85 P ce29596314c84d3cafbda20ce40f83fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a9ac4ade50a34ea894da6837c0a2aa04 in term 0.
I20260926 15:13:49.747375 31435 leader_election.cc:302] T 0a0d0666b51b4c24b213cbe30bb92e85 P a9ac4ade50a34ea894da6837c0a2aa04 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a9ac4ade50a34ea894da6837c0a2aa04, ce29596314c84d3cafbda20ce40f83fd; no voters:
I20260926 15:13:49.747736 31700 raft_consensus.cc:2781] T 0a0d0666b51b4c24b213cbe30bb92e85 P a9ac4ade50a34ea894da6837c0a2aa04 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 15:13:49.747888 31700 raft_consensus.cc:482] T 0a0d0666b51b4c24b213cbe30bb92e85 P a9ac4ade50a34ea894da6837c0a2aa04 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 15:13:49.747982 31700 raft_consensus.cc:3037] T 0a0d0666b51b4c24b213cbe30bb92e85 P a9ac4ade50a34ea894da6837c0a2aa04 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:13:49.750885 31678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:49.740226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.194:41235 (local address 127.12.34.195:51301)
0926 15:13:49.741328 (+ 1102us) negotiation.cc:107] Waiting for socket to connect
0926 15:13:49.741352 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 15:13:49.741482 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:13:49.742639 (+ 1157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:13:49.742647 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:13:49.742681 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:13:49.743023 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:13:49.743054 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:13:49.744876 (+ 1822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:13:49.744884 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:13:49.748513 (+ 3629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:13:49.748530 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:13:49.750147 (+ 1617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:13:49.750153 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:13:49.750329 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:13:49.750365 (+ 36us) client_negotiation.cc:769] Sending connection context
0926 15:13:49.750463 (+ 98us) client_negotiation.cc:240] Negotiation successful
0926 15:13:49.750592 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":947,"mutex_wait_us":1}
I20260926 15:13:49.750893 31689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:49.741880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.195:51301 (local address 127.12.34.194:41235)
0926 15:13:49.742145 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0926 15:13:49.742153 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 15:13:49.742175 (+ 22us) server_negotiation.cc:407] Connection header received
0926 15:13:49.742243 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:13:49.742249 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:13:49.742343 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:13:49.742445 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:13:49.743295 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:49.744716 (+ 1421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:49.748680 (+ 3964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:49.749987 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:49.750142 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:13:49.750204 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 15:13:49.750494 (+ 290us) server_negotiation.cc:299] Negotiation successful
0926 15:13:49.750616 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260926 15:13:49.751068 31700 raft_consensus.cc:504] T 0a0d0666b51b4c24b213cbe30bb92e85 P a9ac4ade50a34ea894da6837c0a2aa04 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9ac4ade50a34ea894da6837c0a2aa04" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 41235 } } peers { permanent_uuid: "ce29596314c84d3cafbda20ce40f83fd" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 36171 } } peers { permanent_uuid: "180820963b6845c1bfe7e9ecde750a3d" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 46673 } }
I20260926 15:13:49.751879 31691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:49.740826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.195:49399 (local address 127.12.34.193:36171)
0926 15:13:49.744224 (+ 3398us) server_negotiation.cc:206] Beginning negotiation
0926 15:13:49.744229 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:13:49.745579 (+ 1350us) server_negotiation.cc:407] Connection header received
0926 15:13:49.745837 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:13:49.745842 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:13:49.745893 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:13:49.745937 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:13:49.746665 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:49.747796 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:49.749737 (+ 1941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:49.750773 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:49.750922 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:13:49.750999 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 15:13:49.751482 (+ 483us) server_negotiation.cc:299] Negotiation successful
0926 15:13:49.751597 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3314,"thread_start_us":3738,"threads_started":1}
I20260926 15:13:49.751890 31705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:49.740778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.193:36171 (local address 127.12.34.195:49399)
0926 15:13:49.745486 (+ 4708us) negotiation.cc:107] Waiting for socket to connect
0926 15:13:49.745505 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 15:13:49.745680 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:13:49.746083 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:13:49.746086 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:13:49.746105 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:13:49.746443 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:13:49.746456 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:13:49.747971 (+ 1515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:13:49.747976 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:13:49.749560 (+ 1584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:13:49.749571 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:13:49.751099 (+ 1528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:13:49.751104 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:13:49.751309 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:13:49.751332 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 15:13:49.751439 (+ 107us) client_negotiation.cc:240] Negotiation successful
0926 15:13:49.751566 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4584,"mutex_wait_us":1,"thread_start_us":4393,"threads_started":1}
I20260926 15:13:49.752096 31501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a0d0666b51b4c24b213cbe30bb92e85" candidate_uuid: "180820963b6845c1bfe7e9ecde750a3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9ac4ade50a34ea894da6837c0a2aa04" is_pre_election: true
I20260926 15:13:49.752362 31700 leader_election.cc:288] T 0a0d0666b51b4c24b213cbe30bb92e85 P a9ac4ade50a34ea894da6837c0a2aa04 [CANDIDATE]: Term 1 election: Requested vote from peers ce29596314c84d3cafbda20ce40f83fd (127.12.34.193:36171), 180820963b6845c1bfe7e9ecde750a3d (127.12.34.195:46673)
I20260926 15:13:49.752447 31631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a0d0666b51b4c24b213cbe30bb92e85" candidate_uuid: "a9ac4ade50a34ea894da6837c0a2aa04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "180820963b6845c1bfe7e9ecde750a3d"
I20260926 15:13:49.752544 31501 raft_consensus.cc:2379] T 0a0d0666b51b4c24b213cbe30bb92e85 P a9ac4ade50a34ea894da6837c0a2aa04 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 180820963b6845c1bfe7e9ecde750a3d in current term 1: Already voted for candidate a9ac4ade50a34ea894da6837c0a2aa04 in this term.
I20260926 15:13:49.752625 31631 raft_consensus.cc:3037] T 0a0d0666b51b4c24b213cbe30bb92e85 P 180820963b6845c1bfe7e9ecde750a3d [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:13:49.752861 31370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a0d0666b51b4c24b213cbe30bb92e85" candidate_uuid: "a9ac4ade50a34ea894da6837c0a2aa04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce29596314c84d3cafbda20ce40f83fd"
I20260926 15:13:49.753105 31370 raft_consensus.cc:3037] T 0a0d0666b51b4c24b213cbe30bb92e85 P ce29596314c84d3cafbda20ce40f83fd [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:13:49.753026 31369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a0d0666b51b4c24b213cbe30bb92e85" candidate_uuid: "180820963b6845c1bfe7e9ecde750a3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce29596314c84d3cafbda20ce40f83fd" is_pre_election: true
I20260926 15:13:49.753926 31565 leader_election.cc:302] T 0a0d0666b51b4c24b213cbe30bb92e85 P 180820963b6845c1bfe7e9ecde750a3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 180820963b6845c1bfe7e9ecde750a3d; no voters: a9ac4ade50a34ea894da6837c0a2aa04, ce29596314c84d3cafbda20ce40f83fd
I20260926 15:13:49.755993 31631 raft_consensus.cc:2454] T 0a0d0666b51b4c24b213cbe30bb92e85 P 180820963b6845c1bfe7e9ecde750a3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a9ac4ade50a34ea894da6837c0a2aa04 in term 1.
I20260926 15:13:49.756235 31702 raft_consensus.cc:2726] T 0a0d0666b51b4c24b213cbe30bb92e85 P 180820963b6845c1bfe7e9ecde750a3d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 15:13:49.756419 31370 raft_consensus.cc:2454] T 0a0d0666b51b4c24b213cbe30bb92e85 P ce29596314c84d3cafbda20ce40f83fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a9ac4ade50a34ea894da6837c0a2aa04 in term 1.
I20260926 15:13:49.756443 31434 leader_election.cc:302] T 0a0d0666b51b4c24b213cbe30bb92e85 P a9ac4ade50a34ea894da6837c0a2aa04 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 180820963b6845c1bfe7e9ecde750a3d, a9ac4ade50a34ea894da6837c0a2aa04; no voters:
I20260926 15:13:49.756784 31700 raft_consensus.cc:2781] T 0a0d0666b51b4c24b213cbe30bb92e85 P a9ac4ade50a34ea894da6837c0a2aa04 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:13:49.757359 31700 raft_consensus.cc:686] T 0a0d0666b51b4c24b213cbe30bb92e85 P a9ac4ade50a34ea894da6837c0a2aa04 [term 1 LEADER]: Becoming Leader. State: Replica: a9ac4ade50a34ea894da6837c0a2aa04, State: Running, Role: LEADER
I20260926 15:13:49.757833 31700 consensus_queue.cc:237] T 0a0d0666b51b4c24b213cbe30bb92e85 P a9ac4ade50a34ea894da6837c0a2aa04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9ac4ade50a34ea894da6837c0a2aa04" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 41235 } } peers { permanent_uuid: "ce29596314c84d3cafbda20ce40f83fd" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 36171 } } peers { permanent_uuid: "180820963b6845c1bfe7e9ecde750a3d" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 46673 } }
I20260926 15:13:49.762179 31228 catalog_manager.cc:5495] T 0a0d0666b51b4c24b213cbe30bb92e85 P a9ac4ade50a34ea894da6837c0a2aa04 reported cstate change: term changed from 0 to 1, leader changed from <none> to a9ac4ade50a34ea894da6837c0a2aa04 (127.12.34.194). New cstate: current_term: 1 leader_uuid: "a9ac4ade50a34ea894da6837c0a2aa04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9ac4ade50a34ea894da6837c0a2aa04" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 41235 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce29596314c84d3cafbda20ce40f83fd" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 36171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "180820963b6845c1bfe7e9ecde750a3d" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 46673 } health_report { overall_health: UNKNOWN } } }
W20260926 15:13:49.764420 31416 tablet.cc:2369] T 0a0d0666b51b4c24b213cbe30bb92e85 P ce29596314c84d3cafbda20ce40f83fd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 15:13:49.804975 31689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:49.799584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60226 (local address 127.12.34.194:41235)
0926 15:13:49.799777 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0926 15:13:49.799784 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:13:49.799804 (+ 20us) server_negotiation.cc:407] Connection header received
0926 15:13:49.799960 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:13:49.799965 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:13:49.800044 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:13:49.800192 (+ 148us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 15:13:49.800570 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:49.801420 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:49.802802 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:49.803204 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:49.803330 (+ 126us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:13:49.803933 (+ 603us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 15:13:49.804387 (+ 454us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 15:13:49.804564 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0926 15:13:49.804682 (+ 118us) server_negotiation.cc:299] Negotiation successful
0926 15:13:49.804724 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 15:13:49.812860 31370 raft_consensus.cc:1261] T 0a0d0666b51b4c24b213cbe30bb92e85 P ce29596314c84d3cafbda20ce40f83fd [term 1 FOLLOWER]: Refusing update from remote peer a9ac4ade50a34ea894da6837c0a2aa04: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:13:49.812913 31631 raft_consensus.cc:1261] T 0a0d0666b51b4c24b213cbe30bb92e85 P 180820963b6845c1bfe7e9ecde750a3d [term 1 FOLLOWER]: Refusing update from remote peer a9ac4ade50a34ea894da6837c0a2aa04: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:13:49.813867 31700 consensus_queue.cc:1035] T 0a0d0666b51b4c24b213cbe30bb92e85 P a9ac4ade50a34ea894da6837c0a2aa04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "180820963b6845c1bfe7e9ecde750a3d" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 46673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:13:49.814240 31707 consensus_queue.cc:1035] T 0a0d0666b51b4c24b213cbe30bb92e85 P a9ac4ade50a34ea894da6837c0a2aa04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce29596314c84d3cafbda20ce40f83fd" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 36171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:13:49.825784 31712 mvcc.cc:205] Tried to move back new op lower bound from 7333624339703140352 to 7333624339491864576. Current Snapshot: MvccSnapshot[applied={T|T < 7333624339703140352}]
I20260926 15:13:49.827519 31710 mvcc.cc:205] Tried to move back new op lower bound from 7333624339703140352 to 7333624339491864576. Current Snapshot: MvccSnapshot[applied={T|T < 7333624339703140352}]
I20260926 15:13:49.842999 31228 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46530:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 15:13:49.843259 31228 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46530:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 15:13:49.846585 31228 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 26844ad8ea554d3e8debbf752c347c9a: Sending DeleteTablet for 3 replicas of tablet 0a0d0666b51b4c24b213cbe30bb92e85
I20260926 15:13:49.847657 31481 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a0d0666b51b4c24b213cbe30bb92e85 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:13:49 UTC) from {username='slave'} at 127.0.0.1:60220
I20260926 15:13:49.847731 31350 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a0d0666b51b4c24b213cbe30bb92e85 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:13:49 UTC) from {username='slave'} at 127.0.0.1:43892
I20260926 15:13:49.848819 12427 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 31288
I20260926 15:13:49.848779 31611 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a0d0666b51b4c24b213cbe30bb92e85 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:13:49 UTC) from {username='slave'} at 127.0.0.1:35534
I20260926 15:13:49.848886 31721 tablet_replica.cc:330] T 0a0d0666b51b4c24b213cbe30bb92e85 P a9ac4ade50a34ea894da6837c0a2aa04: stopping tablet replica
I20260926 15:13:49.849670 31721 raft_consensus.cc:2229] T 0a0d0666b51b4c24b213cbe30bb92e85 P a9ac4ade50a34ea894da6837c0a2aa04 [term 1 LEADER]: Raft consensus shutting down.
I20260926 15:13:49.850411 31721 raft_consensus.cc:2258] T 0a0d0666b51b4c24b213cbe30bb92e85 P a9ac4ade50a34ea894da6837c0a2aa04 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:13:49.851207 31722 tablet_replica.cc:330] T 0a0d0666b51b4c24b213cbe30bb92e85 P ce29596314c84d3cafbda20ce40f83fd: stopping tablet replica
I20260926 15:13:49.851670 31722 raft_consensus.cc:2229] T 0a0d0666b51b4c24b213cbe30bb92e85 P ce29596314c84d3cafbda20ce40f83fd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 15:13:49.852218 31722 raft_consensus.cc:2258] T 0a0d0666b51b4c24b213cbe30bb92e85 P ce29596314c84d3cafbda20ce40f83fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:13:49.853307 31723 tablet_replica.cc:330] T 0a0d0666b51b4c24b213cbe30bb92e85 P 180820963b6845c1bfe7e9ecde750a3d: stopping tablet replica
I20260926 15:13:49.853780 31723 raft_consensus.cc:2229] T 0a0d0666b51b4c24b213cbe30bb92e85 P 180820963b6845c1bfe7e9ecde750a3d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 15:13:49.854367 31723 raft_consensus.cc:2258] T 0a0d0666b51b4c24b213cbe30bb92e85 P 180820963b6845c1bfe7e9ecde750a3d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:13:49.854374 31721 ts_tablet_manager.cc:1905] T 0a0d0666b51b4c24b213cbe30bb92e85 P a9ac4ade50a34ea894da6837c0a2aa04: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:13:49.856923 31723 ts_tablet_manager.cc:1905] T 0a0d0666b51b4c24b213cbe30bb92e85 P 180820963b6845c1bfe7e9ecde750a3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:13:49.861677 31721 ts_tablet_manager.cc:1918] T 0a0d0666b51b4c24b213cbe30bb92e85 P a9ac4ade50a34ea894da6837c0a2aa04: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 15:13:49.863451 31722 ts_tablet_manager.cc:1905] T 0a0d0666b51b4c24b213cbe30bb92e85 P ce29596314c84d3cafbda20ce40f83fd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:13:49.865773 31723 ts_tablet_manager.cc:1918] T 0a0d0666b51b4c24b213cbe30bb92e85 P 180820963b6845c1bfe7e9ecde750a3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 15:13:49.866063 31723 log.cc:1196] T 0a0d0666b51b4c24b213cbe30bb92e85 P 180820963b6845c1bfe7e9ecde750a3d: Deleting WAL directory at /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790435318278681-12427-0/minicluster-data/ts-2/wal/wals/0a0d0666b51b4c24b213cbe30bb92e85
I20260926 15:13:49.866776 31723 ts_tablet_manager.cc:1939] T 0a0d0666b51b4c24b213cbe30bb92e85 P 180820963b6845c1bfe7e9ecde750a3d: Deleting consensus metadata
I20260926 15:13:49.869225 31213 catalog_manager.cc:4841] TS 180820963b6845c1bfe7e9ecde750a3d (127.12.34.195:46673): tablet 0a0d0666b51b4c24b213cbe30bb92e85 (table test-table [id=11e3ca4a4ad84b41820873ad9525b1c0]) successfully deleted
I20260926 15:13:49.869870 31721 log.cc:1196] T 0a0d0666b51b4c24b213cbe30bb92e85 P a9ac4ade50a34ea894da6837c0a2aa04: Deleting WAL directory at /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790435318278681-12427-0/minicluster-data/ts-1/wal/wals/0a0d0666b51b4c24b213cbe30bb92e85
I20260926 15:13:49.870589 31721 ts_tablet_manager.cc:1939] T 0a0d0666b51b4c24b213cbe30bb92e85 P a9ac4ade50a34ea894da6837c0a2aa04: Deleting consensus metadata
I20260926 15:13:49.871327 31722 ts_tablet_manager.cc:1918] T 0a0d0666b51b4c24b213cbe30bb92e85 P ce29596314c84d3cafbda20ce40f83fd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 15:13:49.871552 31722 log.cc:1196] T 0a0d0666b51b4c24b213cbe30bb92e85 P ce29596314c84d3cafbda20ce40f83fd: Deleting WAL directory at /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790435318278681-12427-0/minicluster-data/ts-0/wal/wals/0a0d0666b51b4c24b213cbe30bb92e85
I20260926 15:13:49.872320 31722 ts_tablet_manager.cc:1939] T 0a0d0666b51b4c24b213cbe30bb92e85 P ce29596314c84d3cafbda20ce40f83fd: Deleting consensus metadata
I20260926 15:13:49.872310 31213 catalog_manager.cc:4841] TS a9ac4ade50a34ea894da6837c0a2aa04 (127.12.34.194:41235): tablet 0a0d0666b51b4c24b213cbe30bb92e85 (table test-table [id=11e3ca4a4ad84b41820873ad9525b1c0]) successfully deleted
I20260926 15:13:49.874089 31214 catalog_manager.cc:4841] TS ce29596314c84d3cafbda20ce40f83fd (127.12.34.193:36171): tablet 0a0d0666b51b4c24b213cbe30bb92e85 (table test-table [id=11e3ca4a4ad84b41820873ad9525b1c0]) successfully deleted
I20260926 15:13:49.876699 31691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:49.849925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43906 (local address 127.12.34.193:36171)
0926 15:13:49.850144 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0926 15:13:49.850152 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 15:13:49.869755 (+ 19603us) server_negotiation.cc:407] Connection header received
0926 15:13:49.870031 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:13:49.870038 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:13:49.870175 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:13:49.870291 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:13:49.870986 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:49.872132 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:49.873520 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:49.874035 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:49.874163 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:13:49.875336 (+ 1173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:13:49.875345 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:13:49.875358 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:13:49.876002 (+ 644us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:13:49.876055 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:13:49.876075 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:13:49.876169 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 15:13:49.876334 (+ 165us) server_negotiation.cc:299] Negotiation successful
0926 15:13:49.876411 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 15:13:49.877429 31410 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:13:49.976310 12427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8uoJGv/build/asan/bin/kudu with pid 31288
I20260926 15:13:50.001741 12427 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 31419
I20260926 15:13:50.009536 31689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:50.002481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60230 (local address 127.12.34.194:41235)
0926 15:13:50.002670 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0926 15:13:50.002676 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:13:50.002697 (+ 21us) server_negotiation.cc:407] Connection header received
0926 15:13:50.002824 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:13:50.002831 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:13:50.002946 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:13:50.003069 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:13:50.003383 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:50.004550 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:50.005980 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:50.006694 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:50.006821 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:13:50.007908 (+ 1087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:13:50.007918 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:13:50.007935 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:13:50.008708 (+ 773us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:13:50.008778 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:13:50.008802 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:13:50.008918 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0926 15:13:50.009048 (+ 130us) server_negotiation.cc:299] Negotiation successful
0926 15:13:50.009171 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 15:13:50.010250 31541 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:13:50.131264 12427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8uoJGv/build/asan/bin/kudu with pid 31419
I20260926 15:13:50.151465 12427 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 31549
I20260926 15:13:50.159127 31693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:50.152439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35540 (local address 127.12.34.195:46673)
0926 15:13:50.152719 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0926 15:13:50.152726 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:13:50.152848 (+ 122us) server_negotiation.cc:407] Connection header received
0926 15:13:50.153260 (+ 412us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:13:50.153265 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:13:50.153376 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:13:50.153684 (+ 308us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:13:50.154124 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:50.154864 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:50.156147 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:50.156538 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:50.156667 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:13:50.157786 (+ 1119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:13:50.157794 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:13:50.157810 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:13:50.158379 (+ 569us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:13:50.158426 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:13:50.158445 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:13:50.158567 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0926 15:13:50.158688 (+ 121us) server_negotiation.cc:299] Negotiation successful
0926 15:13:50.158812 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260926 15:13:50.159996 31671 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:13:50.250540 12427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8uoJGv/build/asan/bin/kudu with pid 31549
I20260926 15:13:50.267581 12427 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 31198
I20260926 15:13:50.268740 31259 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:13:50.378230 12427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8uoJGv/build/asan/bin/kudu with pid 31198
2026-09-26T15:13:50Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (2941 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435318278681-12427-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 15:13:50 dist-test-slave-865v krb5kdc[31731](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 15:13:50 dist-test-slave-865v krb5kdc[31731](info): set up 2 sockets
Sep 26 15:13:50 dist-test-slave-865v krb5kdc[31731](info): commencing operation
krb5kdc: starting...
W20260926 15:13:51.509446 12427 mini_kdc.cc:121] Time spent starting KDC: real 1.090s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435318278681-12427-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435318278681-12427-0/krb5kdc/test-user.keytab.
Sep 26 15:13:51 dist-test-slave-865v krb5kdc[31731](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435631, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T15:13:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T15:13:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.34.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.34.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.34.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.254.keytab.
Entry for principal kudu/127.12.34.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.254.keytab.
I20260926 15:13:51.748930 12427 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8uoJGv/build/asan/bin/kudu
/tmp/dist-test-task8uoJGv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435318278681-12427-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435318278681-12427-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435318278681-12427-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.34.254:43809
--webserver_interface=127.12.34.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.254.keytab
--principal=kudu/127.12.34.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.34.212:42235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.34.254:43809
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435318278681-12427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435318278681-12427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435318278681-12427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:13:52.069612 31744 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:13:52.069928 31744 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:13:52.070094 31744 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:13:52.077780 31744 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:13:52.077927 31744 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:13:52.078025 31744 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:13:52.078128 31744 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:13:52.078217 31744 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:13:52.088111 31744 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.34.212:42235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435318278681-12427-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435318278681-12427-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.34.254:43809
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.254.keytab
--principal=kudu/127.12.34.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.34.254:43809
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.34.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435318278681-12427-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:13:52.089529 31744 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:13:52.090781 31744 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:13:52.104144 31749 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:13:52.104195 31750 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:13:52.104802 31752 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:13:52.105327 31744 server_base.cc:1034] running on GCE node
I20260926 15:13:52.106269 31744 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:13:52.107791 31744 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:13:52.109153 31744 hybrid_clock.cc:648] HybridClock initialized: now 1790435632109038 us; error 44 us; skew 500 ppm
I20260926 15:13:52.109568 31744 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:13:52 dist-test-slave-865v krb5kdc[31731](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435632, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:13:52.112767 31744 init.cc:377] Logged in from keytab as kudu/127.12.34.254@KRBTEST.COM (short username kudu)
I20260926 15:13:52.114313 31744 webserver.cc:460] Webserver started at http://127.12.34.254:35047/ using document root <none> and password file <none>
I20260926 15:13:52.114988 31744 fs_manager.cc:359] Metadata directory not provided
I20260926 15:13:52.115134 31744 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:13:52.115427 31744 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:13:52.118412 31744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435318278681-12427-0/minicluster-data/master-0/data/instance:
uuid: "d91b3e5a843e42668bd8b3f545705821"
format_stamp: "Formatted at 2026-09-26 15:13:52 on dist-test-slave-865v"
server_key: "01263054367146134f5e88bacced3c52"
server_key_iv: "e82de5633aa8bb84c09a81f32ecfe33e"
server_key_version: "encryptionkey@0"
I20260926 15:13:52.119184 31744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435318278681-12427-0/minicluster-data/master-0/wal/instance:
uuid: "d91b3e5a843e42668bd8b3f545705821"
format_stamp: "Formatted at 2026-09-26 15:13:52 on dist-test-slave-865v"
server_key: "01263054367146134f5e88bacced3c52"
server_key_iv: "e82de5633aa8bb84c09a81f32ecfe33e"
server_key_version: "encryptionkey@0"
I20260926 15:13:52.124111 31744 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260926 15:13:52.127439 31759 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:13:52.128616 31744 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 15:13:52.128803 31744 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435318278681-12427-0/minicluster-data/master-0/data,/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435318278681-12427-0/minicluster-data/master-0/wal
uuid: "d91b3e5a843e42668bd8b3f545705821"
format_stamp: "Formatted at 2026-09-26 15:13:52 on dist-test-slave-865v"
server_key: "01263054367146134f5e88bacced3c52"
server_key_iv: "e82de5633aa8bb84c09a81f32ecfe33e"
server_key_version: "encryptionkey@0"
I20260926 15:13:52.128986 31744 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435318278681-12427-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435318278681-12427-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435318278681-12427-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:13:52.152200 31744 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:13:52.156301 31744 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:13:52.156594 31744 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:13:52.175589 31744 rpc_server.cc:307] RPC server started. Bound to: 127.12.34.254:43809
I20260926 15:13:52.175613 31810 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.34.254:43809 every 8 connection(s)
I20260926 15:13:52.177040 31744 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
I20260926 15:13:52.180811 31811 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:13:52.180938 12427 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8uoJGv/build/asan/bin/kudu as pid 31744
I20260926 15:13:52.181139 12427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435318278681-12427-0/minicluster-data/master-0/wal/instance
I20260926 15:13:52.181705 12427 external_mini_cluster.cc:1428] Setting key 2b0c1a7e1c5b6c396574a290e6c71678
I20260926 15:13:52.195375 31811 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d91b3e5a843e42668bd8b3f545705821: Bootstrap starting.
Sep 26 15:13:52 dist-test-slave-865v krb5kdc[31731](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435631, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.34.254@KRBTEST.COM
I20260926 15:13:52.198415 31811 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d91b3e5a843e42668bd8b3f545705821: Neither blocks nor log segments found. Creating new log.
I20260926 15:13:52.199441 31811 log.cc:824] T 00000000000000000000000000000000 P d91b3e5a843e42668bd8b3f545705821: Log is configured to *not* fsync() on all Append() calls
I20260926 15:13:52.202441 31811 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d91b3e5a843e42668bd8b3f545705821: No bootstrap required, opened a new log
I20260926 15:13:52.204303 31814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:52.183489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60968 (local address 127.12.34.254:43809)
0926 15:13:52.184314 (+ 825us) server_negotiation.cc:206] Beginning negotiation
0926 15:13:52.184324 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 15:13:52.184363 (+ 39us) server_negotiation.cc:407] Connection header received
0926 15:13:52.185916 (+ 1553us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:13:52.185937 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:13:52.186295 (+ 358us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:13:52.186629 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:13:52.188356 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:52.189672 (+ 1316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:52.191416 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:52.192118 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:52.192343 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:13:52.196950 (+ 4607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:13:52.196957 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:13:52.196969 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:13:52.197003 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:13:52.200381 (+ 3378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:13:52.201137 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:13:52.201147 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:13:52.201158 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:13:52.201326 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:13:52.201705 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:13:52.201712 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:13:52.201715 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:13:52.202223 (+ 508us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:13:52.202536 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0926 15:13:52.202944 (+ 408us) server_negotiation.cc:299] Negotiation successful
0926 15:13:52.203374 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":616,"thread_start_us":401,"threads_started":1}
I20260926 15:13:52.210359 31811 raft_consensus.cc:348] T 00000000000000000000000000000000 P d91b3e5a843e42668bd8b3f545705821 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d91b3e5a843e42668bd8b3f545705821" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 43809 } }
I20260926 15:13:52.210767 31811 raft_consensus.cc:374] T 00000000000000000000000000000000 P d91b3e5a843e42668bd8b3f545705821 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:13:52.210881 31811 raft_consensus.cc:729] T 00000000000000000000000000000000 P d91b3e5a843e42668bd8b3f545705821 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d91b3e5a843e42668bd8b3f545705821, State: Initialized, Role: FOLLOWER
I20260926 15:13:52.211465 31811 consensus_queue.cc:260] T 00000000000000000000000000000000 P d91b3e5a843e42668bd8b3f545705821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d91b3e5a843e42668bd8b3f545705821" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 43809 } }
I20260926 15:13:52.211709 31811 raft_consensus.cc:388] T 00000000000000000000000000000000 P d91b3e5a843e42668bd8b3f545705821 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 15:13:52.211853 31811 raft_consensus.cc:482] T 00000000000000000000000000000000 P d91b3e5a843e42668bd8b3f545705821 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 15:13:52.212023 31811 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d91b3e5a843e42668bd8b3f545705821 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:13:52.214591 31811 raft_consensus.cc:504] T 00000000000000000000000000000000 P d91b3e5a843e42668bd8b3f545705821 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d91b3e5a843e42668bd8b3f545705821" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 43809 } }
I20260926 15:13:52.215080 31811 leader_election.cc:302] T 00000000000000000000000000000000 P d91b3e5a843e42668bd8b3f545705821 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d91b3e5a843e42668bd8b3f545705821; no voters:
I20260926 15:13:52.215905 31811 leader_election.cc:288] T 00000000000000000000000000000000 P d91b3e5a843e42668bd8b3f545705821 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 15:13:52.215989 31816 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d91b3e5a843e42668bd8b3f545705821 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:13:52.216811 31816 raft_consensus.cc:686] T 00000000000000000000000000000000 P d91b3e5a843e42668bd8b3f545705821 [term 1 LEADER]: Becoming Leader. State: Replica: d91b3e5a843e42668bd8b3f545705821, State: Running, Role: LEADER
I20260926 15:13:52.217482 31816 consensus_queue.cc:237] T 00000000000000000000000000000000 P d91b3e5a843e42668bd8b3f545705821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d91b3e5a843e42668bd8b3f545705821" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 43809 } }
I20260926 15:13:52.218278 31811 sys_catalog.cc:564] T 00000000000000000000000000000000 P d91b3e5a843e42668bd8b3f545705821 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:13:52.220167 31818 sys_catalog.cc:455] T 00000000000000000000000000000000 P d91b3e5a843e42668bd8b3f545705821 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d91b3e5a843e42668bd8b3f545705821" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d91b3e5a843e42668bd8b3f545705821" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 43809 } } }
I20260926 15:13:52.220813 31818 sys_catalog.cc:458] T 00000000000000000000000000000000 P d91b3e5a843e42668bd8b3f545705821 [sys.catalog]: This master's current role is: LEADER
I20260926 15:13:52.225929 31822 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:13:52.225821 31817 sys_catalog.cc:455] T 00000000000000000000000000000000 P d91b3e5a843e42668bd8b3f545705821 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d91b3e5a843e42668bd8b3f545705821. Latest consensus state: current_term: 1 leader_uuid: "d91b3e5a843e42668bd8b3f545705821" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d91b3e5a843e42668bd8b3f545705821" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 43809 } } }
I20260926 15:13:52.226233 31817 sys_catalog.cc:458] T 00000000000000000000000000000000 P d91b3e5a843e42668bd8b3f545705821 [sys.catalog]: This master's current role is: LEADER
I20260926 15:13:52.231622 31822 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 15:13:52.240361 31822 catalog_manager.cc:1347] Generated new cluster ID: 85f1ed2f10ff401fb311635dc39181f1
I20260926 15:13:52.240479 31822 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 15:13:52.259992 31822 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 15:13:52.262641 31822 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:13:52.281780 31822 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d91b3e5a843e42668bd8b3f545705821: Generated new TSK 0
I20260926 15:13:52.283023 31822 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.34.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.34.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.34.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.193.keytab.
Entry for principal kudu/127.12.34.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.193.keytab.
I20260926 15:13:52.356768 12427 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8uoJGv/build/asan/bin/kudu
/tmp/dist-test-task8uoJGv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435318278681-12427-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435318278681-12427-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435318278681-12427-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435318278681-12427-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.34.193:0
--local_ip_for_outbound_sockets=127.12.34.193
--webserver_interface=127.12.34.193
--webserver_port=0
--tserver_master_addrs=127.12.34.254:43809
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.193.keytab
--principal=kudu/127.12.34.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.34.212:42235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435318278681-12427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435318278681-12427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435318278681-12427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:13:52.721776 31837 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:13:52.722034 31837 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:13:52.722127 31837 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:13:52.729508 31837 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:13:52.729614 31837 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:13:52.729729 31837 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.34.193
I20260926 15:13:52.739189 31837 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.34.212:42235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435318278681-12427-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435318278681-12427-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.193.keytab
--principal=kudu/127.12.34.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.34.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435318278681-12427-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.34.193
--webserver_port=0
--tserver_master_addrs=127.12.34.254:43809
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.34.193
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435318278681-12427-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:13:52.740523 31837 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:13:52.741837 31837 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:13:52.754035 31842 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:13:52.755223 31843 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:13:52.755650 31845 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:13:52.756556 31837 server_base.cc:1034] running on GCE node
I20260926 15:13:52.757256 31837 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:13:52.758241 31837 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:13:52.759588 31837 hybrid_clock.cc:648] HybridClock initialized: now 1790435632759519 us; error 55 us; skew 500 ppm
I20260926 15:13:52.759924 31837 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:13:52 dist-test-slave-865v krb5kdc[31731](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435632, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:13:52.763108 31837 init.cc:377] Logged in from keytab as kudu/127.12.34.193@KRBTEST.COM (short username kudu)
I20260926 15:13:52.764595 31837 webserver.cc:460] Webserver started at http://127.12.34.193:38799/ using document root <none> and password file <none>
I20260926 15:13:52.765360 31837 fs_manager.cc:359] Metadata directory not provided
I20260926 15:13:52.765501 31837 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:13:52.765800 31837 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:13:52.768575 31837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435318278681-12427-0/minicluster-data/ts-0/data/instance:
uuid: "3732c46edbc046aaad375846a0058925"
format_stamp: "Formatted at 2026-09-26 15:13:52 on dist-test-slave-865v"
server_key: "19e2045bc5a1c4e77c963d31f64689d9"
server_key_iv: "3fce929a7a70c8844703d98628a55d24"
server_key_version: "encryptionkey@0"
I20260926 15:13:52.769376 31837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435318278681-12427-0/minicluster-data/ts-0/wal/instance:
uuid: "3732c46edbc046aaad375846a0058925"
format_stamp: "Formatted at 2026-09-26 15:13:52 on dist-test-slave-865v"
server_key: "19e2045bc5a1c4e77c963d31f64689d9"
server_key_iv: "3fce929a7a70c8844703d98628a55d24"
server_key_version: "encryptionkey@0"
I20260926 15:13:52.774760 31837 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 15:13:52.778275 31852 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:13:52.779577 31837 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 15:13:52.779740 31837 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435318278681-12427-0/minicluster-data/ts-0/data,/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435318278681-12427-0/minicluster-data/ts-0/wal
uuid: "3732c46edbc046aaad375846a0058925"
format_stamp: "Formatted at 2026-09-26 15:13:52 on dist-test-slave-865v"
server_key: "19e2045bc5a1c4e77c963d31f64689d9"
server_key_iv: "3fce929a7a70c8844703d98628a55d24"
server_key_version: "encryptionkey@0"
I20260926 15:13:52.779942 31837 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435318278681-12427-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435318278681-12427-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435318278681-12427-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:13:52.815044 31837 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:13:52.819464 31837 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:13:52.819783 31837 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:13:52.820725 31837 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:13:52.822032 31837 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:13:52.822111 31837 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:13:52.822232 31837 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:13:52.822293 31837 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:13:52.855944 31837 rpc_server.cc:307] RPC server started. Bound to: 127.12.34.193:38607
I20260926 15:13:52.856041 31964 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.34.193:38607 every 8 connection(s)
I20260926 15:13:52.857237 31837 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435318278681-12427-0/minicluster-data/ts-0/data/info.pb
I20260926 15:13:52.864837 12427 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8uoJGv/build/asan/bin/kudu as pid 31837
I20260926 15:13:52.865032 12427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435318278681-12427-0/minicluster-data/ts-0/wal/instance
I20260926 15:13:52.865545 12427 external_mini_cluster.cc:1428] Setting key 33c82e71ef8beecd56bc171bdc6ca3f3
Sep 26 15:13:52 dist-test-slave-865v krb5kdc[31731](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435632, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.193@KRBTEST.COM for kudu/127.12.34.254@KRBTEST.COM
I20260926 15:13:52.877058 31967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:52.859741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.193:47461 (local address 127.12.34.254:43809)
0926 15:13:52.860207 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0926 15:13:52.860211 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:13:52.861109 (+ 898us) server_negotiation.cc:407] Connection header received
0926 15:13:52.862929 (+ 1820us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:13:52.862933 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:13:52.862998 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:13:52.863094 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:13:52.864464 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:52.865337 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:52.867218 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:52.867680 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:52.867782 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:13:52.871720 (+ 3938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:13:52.871725 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:13:52.871728 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:13:52.871758 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:13:52.874192 (+ 2434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:13:52.874878 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:13:52.874884 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:13:52.874890 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:13:52.875005 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:13:52.875404 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:13:52.875410 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:13:52.875413 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:13:52.875664 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:13:52.875826 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0926 15:13:52.876494 (+ 668us) server_negotiation.cc:299] Negotiation successful
0926 15:13:52.876710 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":241,"threads_started":1}
I20260926 15:13:52.877966 31968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:52.860285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:43809 (local address 127.12.34.193:47461)
0926 15:13:52.860931 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0926 15:13:52.860972 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 15:13:52.862669 (+ 1697us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:13:52.863285 (+ 616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:13:52.863292 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:13:52.863794 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:13:52.864312 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:13:52.864327 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:13:52.865483 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:13:52.865489 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:13:52.867041 (+ 1552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:13:52.867053 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:13:52.867909 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:13:52.867916 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:13:52.868203 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:13:52.868900 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:13:52.868934 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:13:52.871499 (+ 2565us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:13:52.874330 (+ 2831us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:13:52.874339 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:13:52.874354 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:13:52.874743 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:13:52.875111 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:13:52.875117 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:13:52.875120 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:13:52.875295 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:13:52.875795 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:13:52.875804 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:13:52.876107 (+ 303us) client_negotiation.cc:769] Sending connection context
0926 15:13:52.876358 (+ 251us) client_negotiation.cc:240] Negotiation successful
0926 15:13:52.876816 (+ 458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":250,"threads_started":1}
I20260926 15:13:52.879521 31965 heartbeater.cc:347] Connected to a master server at 127.12.34.254:43809
I20260926 15:13:52.879801 31965 heartbeater.cc:464] Registering TS with master...
I20260926 15:13:52.880461 31965 heartbeater.cc:511] Master 127.12.34.254:43809 requested a full tablet report, sending...
I20260926 15:13:52.882716 31776 ts_manager.cc:194] Registered new tserver with Master: 3732c46edbc046aaad375846a0058925 (127.12.34.193:38607)
I20260926 15:13:52.884944 31776 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.34.193@KRBTEST.COM'} at 127.12.34.193:47461
WARNING: no policy specified for kudu/127.12.34.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.34.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.34.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.194.keytab.
Entry for principal kudu/127.12.34.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.194.keytab.
I20260926 15:13:52.921047 12427 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8uoJGv/build/asan/bin/kudu
/tmp/dist-test-task8uoJGv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435318278681-12427-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435318278681-12427-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435318278681-12427-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435318278681-12427-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.34.194:0
--local_ip_for_outbound_sockets=127.12.34.194
--webserver_interface=127.12.34.194
--webserver_port=0
--tserver_master_addrs=127.12.34.254:43809
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.194.keytab
--principal=kudu/127.12.34.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.34.212:42235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435318278681-12427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435318278681-12427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435318278681-12427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:13:53.245631 31971 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:13:53.245889 31971 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:13:53.245985 31971 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:13:53.252955 31971 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:13:53.253057 31971 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:13:53.253238 31971 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.34.194
I20260926 15:13:53.262573 31971 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.34.212:42235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435318278681-12427-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435318278681-12427-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.194.keytab
--principal=kudu/127.12.34.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.34.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435318278681-12427-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.34.194
--webserver_port=0
--tserver_master_addrs=127.12.34.254:43809
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.34.194
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435318278681-12427-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:13:53.263720 31971 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:13:53.264910 31971 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:13:53.277347 31977 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:13:53.277493 31976 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:13:53.277781 31979 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:13:53.278594 31971 server_base.cc:1034] running on GCE node
I20260926 15:13:53.279198 31971 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:13:53.280074 31971 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:13:53.281293 31971 hybrid_clock.cc:648] HybridClock initialized: now 1790435633281240 us; error 45 us; skew 500 ppm
I20260926 15:13:53.281585 31971 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:13:53 dist-test-slave-865v krb5kdc[31731](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435633, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:13:53.290498 31971 init.cc:377] Logged in from keytab as kudu/127.12.34.194@KRBTEST.COM (short username kudu)
I20260926 15:13:53.292062 31971 webserver.cc:460] Webserver started at http://127.12.34.194:39967/ using document root <none> and password file <none>
I20260926 15:13:53.292740 31971 fs_manager.cc:359] Metadata directory not provided
I20260926 15:13:53.292889 31971 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:13:53.293272 31971 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:13:53.296074 31971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435318278681-12427-0/minicluster-data/ts-1/data/instance:
uuid: "d68b0d411d1b4e68a77b56b8e5a7d0ad"
format_stamp: "Formatted at 2026-09-26 15:13:53 on dist-test-slave-865v"
server_key: "8f74e4cc1056def13f8cad4ad797cb67"
server_key_iv: "6fa7db56f1e846b5bebd31cd11b5ddd4"
server_key_version: "encryptionkey@0"
I20260926 15:13:53.297124 31971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435318278681-12427-0/minicluster-data/ts-1/wal/instance:
uuid: "d68b0d411d1b4e68a77b56b8e5a7d0ad"
format_stamp: "Formatted at 2026-09-26 15:13:53 on dist-test-slave-865v"
server_key: "8f74e4cc1056def13f8cad4ad797cb67"
server_key_iv: "6fa7db56f1e846b5bebd31cd11b5ddd4"
server_key_version: "encryptionkey@0"
I20260926 15:13:53.302434 31971 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260926 15:13:53.305917 31986 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:13:53.307322 31971 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 15:13:53.307507 31971 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435318278681-12427-0/minicluster-data/ts-1/data,/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435318278681-12427-0/minicluster-data/ts-1/wal
uuid: "d68b0d411d1b4e68a77b56b8e5a7d0ad"
format_stamp: "Formatted at 2026-09-26 15:13:53 on dist-test-slave-865v"
server_key: "8f74e4cc1056def13f8cad4ad797cb67"
server_key_iv: "6fa7db56f1e846b5bebd31cd11b5ddd4"
server_key_version: "encryptionkey@0"
I20260926 15:13:53.307716 31971 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435318278681-12427-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435318278681-12427-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435318278681-12427-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:13:53.327606 31971 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:13:53.332141 31971 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:13:53.332418 31971 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:13:53.333456 31971 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:13:53.334801 31971 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:13:53.334901 31971 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:13:53.335057 31971 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:13:53.335129 31971 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:13:53.368412 31971 rpc_server.cc:307] RPC server started. Bound to: 127.12.34.194:38529
I20260926 15:13:53.368458 32098 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.34.194:38529 every 8 connection(s)
I20260926 15:13:53.369727 31971 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435318278681-12427-0/minicluster-data/ts-1/data/info.pb
I20260926 15:13:53.374531 12427 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8uoJGv/build/asan/bin/kudu as pid 31971
I20260926 15:13:53.374737 12427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435318278681-12427-0/minicluster-data/ts-1/wal/instance
I20260926 15:13:53.375180 12427 external_mini_cluster.cc:1428] Setting key a55ecee63a7cf4db15a68760fdbde14d
Sep 26 15:13:53 dist-test-slave-865v krb5kdc[31731](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435633, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.194@KRBTEST.COM for kudu/127.12.34.254@KRBTEST.COM
I20260926 15:13:53.390370 31967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:53.372214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.194:56319 (local address 127.12.34.254:43809)
0926 15:13:53.372423 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0926 15:13:53.372428 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:13:53.373549 (+ 1121us) server_negotiation.cc:407] Connection header received
0926 15:13:53.375485 (+ 1936us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:13:53.375491 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:13:53.375585 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:13:53.375704 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:13:53.377068 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:53.377788 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:53.379505 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:53.379939 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:53.380074 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:13:53.383664 (+ 3590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:13:53.383669 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:13:53.383672 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:13:53.383704 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:13:53.387018 (+ 3314us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:13:53.387991 (+ 973us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:13:53.387995 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:13:53.387997 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:13:53.388130 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:13:53.388629 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:13:53.388632 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:13:53.388634 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:13:53.388932 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:13:53.389046 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0926 15:13:53.389955 (+ 909us) server_negotiation.cc:299] Negotiation successful
0926 15:13:53.390128 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260926 15:13:53.391778 32101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:53.372650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:43809 (local address 127.12.34.194:56319)
0926 15:13:53.373398 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0926 15:13:53.373440 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 15:13:53.375200 (+ 1760us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:13:53.375915 (+ 715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:13:53.375922 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:13:53.376406 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:13:53.376908 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:13:53.376925 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:13:53.377995 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:13:53.377999 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:13:53.379351 (+ 1352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:13:53.379363 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:13:53.380158 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:13:53.380165 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:13:53.380385 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:13:53.381053 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:13:53.381074 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:13:53.383370 (+ 2296us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:13:53.387154 (+ 3784us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:13:53.387166 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:13:53.387185 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:13:53.387795 (+ 610us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:13:53.388252 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:13:53.388257 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:13:53.388272 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:13:53.388515 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:13:53.389058 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:13:53.389069 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:13:53.389532 (+ 463us) client_negotiation.cc:769] Sending connection context
0926 15:13:53.389849 (+ 317us) client_negotiation.cc:240] Negotiation successful
0926 15:13:53.390391 (+ 542us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":265,"threads_started":1}
I20260926 15:13:53.393179 32099 heartbeater.cc:347] Connected to a master server at 127.12.34.254:43809
I20260926 15:13:53.393482 32099 heartbeater.cc:464] Registering TS with master...
I20260926 15:13:53.394191 32099 heartbeater.cc:511] Master 127.12.34.254:43809 requested a full tablet report, sending...
I20260926 15:13:53.395658 31776 ts_manager.cc:194] Registered new tserver with Master: d68b0d411d1b4e68a77b56b8e5a7d0ad (127.12.34.194:38529)
I20260926 15:13:53.396476 31776 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.34.194@KRBTEST.COM'} at 127.12.34.194:56319
WARNING: no policy specified for kudu/127.12.34.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.34.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.34.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.195.keytab.
Entry for principal kudu/127.12.34.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.195.keytab.
I20260926 15:13:53.432571 12427 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8uoJGv/build/asan/bin/kudu
/tmp/dist-test-task8uoJGv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435318278681-12427-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435318278681-12427-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435318278681-12427-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435318278681-12427-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.34.195:0
--local_ip_for_outbound_sockets=127.12.34.195
--webserver_interface=127.12.34.195
--webserver_port=0
--tserver_master_addrs=127.12.34.254:43809
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.195.keytab
--principal=kudu/127.12.34.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.34.212:42235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435318278681-12427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435318278681-12427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435318278681-12427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:13:53.755318 32104 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:13:53.755626 32104 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:13:53.755798 32104 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:13:53.762867 32104 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:13:53.763016 32104 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:13:53.763195 32104 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.34.195
I20260926 15:13:53.772488 32104 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.34.212:42235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435318278681-12427-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435318278681-12427-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.195.keytab
--principal=kudu/127.12.34.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.34.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435318278681-12427-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.34.195
--webserver_port=0
--tserver_master_addrs=127.12.34.254:43809
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.34.195
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435318278681-12427-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:13:53.773738 32104 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:13:53.775004 32104 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:13:53.787843 32110 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:13:53.787868 32109 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:13:53.788335 32112 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:13:53.788743 32104 server_base.cc:1034] running on GCE node
I20260926 15:13:53.789453 32104 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:13:53.790341 32104 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:13:53.791538 32104 hybrid_clock.cc:648] HybridClock initialized: now 1790435633791494 us; error 32 us; skew 500 ppm
I20260926 15:13:53.791854 32104 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:13:53 dist-test-slave-865v krb5kdc[31731](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435633, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:13:53.794875 32104 init.cc:377] Logged in from keytab as kudu/127.12.34.195@KRBTEST.COM (short username kudu)
I20260926 15:13:53.796307 32104 webserver.cc:460] Webserver started at http://127.12.34.195:38917/ using document root <none> and password file <none>
I20260926 15:13:53.796954 32104 fs_manager.cc:359] Metadata directory not provided
I20260926 15:13:53.797134 32104 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:13:53.797457 32104 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:13:53.800216 32104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435318278681-12427-0/minicluster-data/ts-2/data/instance:
uuid: "b9d63fc120704294ad2e422ba6fd74ec"
format_stamp: "Formatted at 2026-09-26 15:13:53 on dist-test-slave-865v"
server_key: "d491818aafe8c3f734e13282f3167e74"
server_key_iv: "f2d4275f2838359dde9adb38e905a095"
server_key_version: "encryptionkey@0"
I20260926 15:13:53.800951 32104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435318278681-12427-0/minicluster-data/ts-2/wal/instance:
uuid: "b9d63fc120704294ad2e422ba6fd74ec"
format_stamp: "Formatted at 2026-09-26 15:13:53 on dist-test-slave-865v"
server_key: "d491818aafe8c3f734e13282f3167e74"
server_key_iv: "f2d4275f2838359dde9adb38e905a095"
server_key_version: "encryptionkey@0"
I20260926 15:13:53.805881 32104 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260926 15:13:53.809175 32119 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:13:53.810354 32104 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 15:13:53.810520 32104 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435318278681-12427-0/minicluster-data/ts-2/data,/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435318278681-12427-0/minicluster-data/ts-2/wal
uuid: "b9d63fc120704294ad2e422ba6fd74ec"
format_stamp: "Formatted at 2026-09-26 15:13:53 on dist-test-slave-865v"
server_key: "d491818aafe8c3f734e13282f3167e74"
server_key_iv: "f2d4275f2838359dde9adb38e905a095"
server_key_version: "encryptionkey@0"
I20260926 15:13:53.810712 32104 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435318278681-12427-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435318278681-12427-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435318278681-12427-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:13:53.833747 32104 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:13:53.837813 32104 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:13:53.838106 32104 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:13:53.839049 32104 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:13:53.840356 32104 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:13:53.840466 32104 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:13:53.840615 32104 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:13:53.840692 32104 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:13:53.874679 32104 rpc_server.cc:307] RPC server started. Bound to: 127.12.34.195:34237
I20260926 15:13:53.874795 32231 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.34.195:34237 every 8 connection(s)
I20260926 15:13:53.875902 32104 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435318278681-12427-0/minicluster-data/ts-2/data/info.pb
I20260926 15:13:53.885558 12427 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8uoJGv/build/asan/bin/kudu as pid 32104
I20260926 15:13:53.885752 12427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435318278681-12427-0/minicluster-data/ts-2/wal/instance
I20260926 15:13:53.886188 12427 external_mini_cluster.cc:1428] Setting key febbaba085c2e9dd1ecb18a8d93c545e
I20260926 15:13:53.887852 31965 heartbeater.cc:503] Master 127.12.34.254:43809 was elected leader, sending a full tablet report...
Sep 26 15:13:53 dist-test-slave-865v krb5kdc[31731](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435633, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.195@KRBTEST.COM for kudu/127.12.34.254@KRBTEST.COM
I20260926 15:13:53.895764 31967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:53.878448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.195:36985 (local address 127.12.34.254:43809)
0926 15:13:53.878625 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0926 15:13:53.878629 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:13:53.879796 (+ 1167us) server_negotiation.cc:407] Connection header received
0926 15:13:53.881644 (+ 1848us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:13:53.881650 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:13:53.881750 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:13:53.881871 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:13:53.883251 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:53.884005 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:53.885884 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:53.886573 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:53.886807 (+ 234us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:13:53.890661 (+ 3854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:13:53.890666 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:13:53.890669 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:13:53.890700 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:13:53.893046 (+ 2346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:13:53.893864 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:13:53.893868 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:13:53.893871 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:13:53.894011 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:13:53.894402 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:13:53.894405 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:13:53.894407 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:13:53.894653 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:13:53.894760 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 15:13:53.895356 (+ 596us) server_negotiation.cc:299] Negotiation successful
0926 15:13:53.895519 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 15:13:53.896600 32234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:53.878883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:43809 (local address 127.12.34.195:36985)
0926 15:13:53.879643 (+ 760us) negotiation.cc:107] Waiting for socket to connect
0926 15:13:53.879679 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 15:13:53.881388 (+ 1709us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:13:53.882077 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:13:53.882085 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:13:53.882573 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:13:53.883082 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:13:53.883098 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:13:53.884159 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:13:53.884169 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:13:53.885599 (+ 1430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:13:53.885614 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:13:53.886876 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:13:53.886886 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:13:53.887156 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:13:53.887862 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:13:53.887884 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:13:53.890472 (+ 2588us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:13:53.893321 (+ 2849us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:13:53.893337 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:13:53.893347 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:13:53.893729 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:13:53.894115 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:13:53.894119 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:13:53.894122 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:13:53.894294 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:13:53.894757 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:13:53.894762 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:13:53.895028 (+ 266us) client_negotiation.cc:769] Sending connection context
0926 15:13:53.895233 (+ 205us) client_negotiation.cc:240] Negotiation successful
0926 15:13:53.895576 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":285,"threads_started":1}
I20260926 15:13:53.897871 32232 heartbeater.cc:347] Connected to a master server at 127.12.34.254:43809
I20260926 15:13:53.898105 32232 heartbeater.cc:464] Registering TS with master...
I20260926 15:13:53.898695 32232 heartbeater.cc:511] Master 127.12.34.254:43809 requested a full tablet report, sending...
I20260926 15:13:53.900017 31775 ts_manager.cc:194] Registered new tserver with Master: b9d63fc120704294ad2e422ba6fd74ec (127.12.34.195:34237)
I20260926 15:13:53.900988 12427 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 15:13:53.901130 31775 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.34.195@KRBTEST.COM'} at 127.12.34.195:36985
I20260926 15:13:53.919281 31967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:53.905649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60972 (local address 127.12.34.254:43809)
0926 15:13:53.906123 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0926 15:13:53.906130 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:13:53.906384 (+ 254us) server_negotiation.cc:407] Connection header received
0926 15:13:53.906591 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:13:53.906596 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:13:53.906693 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:13:53.906804 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:13:53.908242 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:53.908970 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:53.910269 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:53.910654 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:53.910791 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:13:53.913353 (+ 2562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:13:53.913359 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:13:53.913365 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:13:53.913405 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:13:53.916444 (+ 3039us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:13:53.917224 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:13:53.917229 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:13:53.917233 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:13:53.917377 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:13:53.917780 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:13:53.917792 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:13:53.917796 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:13:53.918108 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:13:53.918372 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0926 15:13:53.918742 (+ 370us) server_negotiation.cc:299] Negotiation successful
0926 15:13:53.918977 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120}
I20260926 15:13:53.923214 31775 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60972:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 15:13:53.930471 31775 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 15:13:53.952864 32249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:53.944252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58338 (local address 127.12.34.194:38529)
0926 15:13:53.945211 (+ 959us) server_negotiation.cc:206] Beginning negotiation
0926 15:13:53.945219 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 15:13:53.945247 (+ 28us) server_negotiation.cc:407] Connection header received
0926 15:13:53.945352 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:13:53.945359 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:13:53.945589 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:13:53.945785 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:13:53.946516 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:53.947812 (+ 1296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:53.950047 (+ 2235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:53.951479 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:53.951628 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:13:53.951845 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0926 15:13:53.952365 (+ 520us) server_negotiation.cc:299] Negotiation successful
0926 15:13:53.952521 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":771,"thread_start_us":661,"threads_started":1}
I20260926 15:13:53.952899 32244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:53.943680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.194:38529 (local address 127.0.0.1:58338)
0926 15:13:53.944608 (+ 928us) negotiation.cc:107] Waiting for socket to connect
0926 15:13:53.944622 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 15:13:53.944721 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:13:53.945832 (+ 1111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:13:53.945842 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:13:53.945886 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:13:53.946343 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:13:53.946355 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:13:53.947945 (+ 1590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:13:53.947952 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:13:53.949849 (+ 1897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:13:53.949866 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:13:53.952006 (+ 2140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:13:53.952013 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:13:53.952144 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:13:53.952174 (+ 30us) client_negotiation.cc:769] Sending connection context
0926 15:13:53.952315 (+ 141us) client_negotiation.cc:240] Negotiation successful
0926 15:13:53.952493 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":811,"mutex_wait_us":22,"thread_start_us":1086,"threads_started":1}
I20260926 15:13:53.954286 32245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:53.942479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.195:34237 (local address 127.0.0.1:36614)
0926 15:13:53.943744 (+ 1265us) negotiation.cc:107] Waiting for socket to connect
0926 15:13:53.943780 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 15:13:53.943943 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:13:53.947186 (+ 3243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:13:53.947190 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:13:53.947214 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:13:53.947600 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:13:53.947611 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:13:53.949594 (+ 1983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:13:53.949599 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:13:53.951585 (+ 1986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:13:53.951598 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:13:53.953365 (+ 1767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:13:53.953377 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:13:53.953595 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:13:53.953666 (+ 71us) client_negotiation.cc:769] Sending connection context
0926 15:13:53.953734 (+ 68us) client_negotiation.cc:240] Negotiation successful
0926 15:13:53.953901 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1025,"mutex_wait_us":13,"thread_start_us":1875,"threads_started":1}
I20260926 15:13:53.954341 32247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:53.942990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34522 (local address 127.12.34.193:38607)
0926 15:13:53.945429 (+ 2439us) server_negotiation.cc:206] Beginning negotiation
0926 15:13:53.945436 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:13:53.945459 (+ 23us) server_negotiation.cc:407] Connection header received
0926 15:13:53.945556 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:13:53.945563 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:13:53.945765 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:13:53.945934 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:13:53.947313 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:53.948868 (+ 1555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:53.951436 (+ 2568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:53.952895 (+ 1459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:53.953032 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:13:53.953287 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0926 15:13:53.953841 (+ 554us) server_negotiation.cc:299] Negotiation successful
0926 15:13:53.953994 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2277,"thread_start_us":2155,"threads_started":1}
I20260926 15:13:53.954638 32246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:53.942842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36614 (local address 127.12.34.195:34237)
0926 15:13:53.946589 (+ 3747us) server_negotiation.cc:206] Beginning negotiation
0926 15:13:53.946597 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 15:13:53.946648 (+ 51us) server_negotiation.cc:407] Connection header received
0926 15:13:53.946746 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:13:53.946752 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:13:53.946972 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:13:53.947779 (+ 807us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:13:53.948045 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:53.949459 (+ 1414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:53.951761 (+ 2302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:53.953156 (+ 1395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:53.953897 (+ 741us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:13:53.954102 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0926 15:13:53.954183 (+ 81us) server_negotiation.cc:299] Negotiation successful
0926 15:13:53.954335 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3580,"thread_start_us":3420,"threads_started":1}
I20260926 15:13:53.955263 32248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:53.942676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.193:38607 (local address 127.0.0.1:34522)
0926 15:13:53.944540 (+ 1864us) negotiation.cc:107] Waiting for socket to connect
0926 15:13:53.944557 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 15:13:53.944670 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:13:53.946063 (+ 1393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:13:53.946068 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:13:53.946094 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:13:53.947035 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:13:53.947047 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:13:53.949012 (+ 1965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:13:53.949018 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:13:53.951265 (+ 2247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:13:53.951280 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:13:53.953441 (+ 2161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:13:53.953448 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:13:53.953628 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:13:53.953671 (+ 43us) client_negotiation.cc:769] Sending connection context
0926 15:13:53.953776 (+ 105us) client_negotiation.cc:240] Negotiation successful
0926 15:13:53.954907 (+ 1131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1694,"mutex_wait_us":963,"thread_start_us":686,"threads_started":1}
I20260926 15:13:53.957249 32034 tablet_service.cc:1467] Processing CreateTablet for tablet 5de5b8fb6e674922872517d886003821 (DEFAULT_TABLE table=test-table [id=ec81e74d4f14494bab66287e2119f6c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:13:53.958452 32034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5de5b8fb6e674922872517d886003821. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:13:53.958056 32167 tablet_service.cc:1467] Processing CreateTablet for tablet 5de5b8fb6e674922872517d886003821 (DEFAULT_TABLE table=test-table [id=ec81e74d4f14494bab66287e2119f6c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:13:53.958735 31900 tablet_service.cc:1467] Processing CreateTablet for tablet 5de5b8fb6e674922872517d886003821 (DEFAULT_TABLE table=test-table [id=ec81e74d4f14494bab66287e2119f6c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:13:53.959398 32167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5de5b8fb6e674922872517d886003821. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:13:53.960069 31900 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5de5b8fb6e674922872517d886003821. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:13:53.969655 32250 tablet_bootstrap.cc:492] T 5de5b8fb6e674922872517d886003821 P d68b0d411d1b4e68a77b56b8e5a7d0ad: Bootstrap starting.
I20260926 15:13:53.972659 32250 tablet_bootstrap.cc:654] T 5de5b8fb6e674922872517d886003821 P d68b0d411d1b4e68a77b56b8e5a7d0ad: Neither blocks nor log segments found. Creating new log.
I20260926 15:13:53.973430 32251 tablet_bootstrap.cc:492] T 5de5b8fb6e674922872517d886003821 P b9d63fc120704294ad2e422ba6fd74ec: Bootstrap starting.
I20260926 15:13:53.975198 32250 log.cc:824] T 5de5b8fb6e674922872517d886003821 P d68b0d411d1b4e68a77b56b8e5a7d0ad: Log is configured to *not* fsync() on all Append() calls
I20260926 15:13:53.976938 32251 tablet_bootstrap.cc:654] T 5de5b8fb6e674922872517d886003821 P b9d63fc120704294ad2e422ba6fd74ec: Neither blocks nor log segments found. Creating new log.
I20260926 15:13:53.976529 32252 tablet_bootstrap.cc:492] T 5de5b8fb6e674922872517d886003821 P 3732c46edbc046aaad375846a0058925: Bootstrap starting.
I20260926 15:13:53.978227 32251 log.cc:824] T 5de5b8fb6e674922872517d886003821 P b9d63fc120704294ad2e422ba6fd74ec: Log is configured to *not* fsync() on all Append() calls
I20260926 15:13:53.978391 32250 tablet_bootstrap.cc:492] T 5de5b8fb6e674922872517d886003821 P d68b0d411d1b4e68a77b56b8e5a7d0ad: No bootstrap required, opened a new log
I20260926 15:13:53.978657 32250 ts_tablet_manager.cc:1397] T 5de5b8fb6e674922872517d886003821 P d68b0d411d1b4e68a77b56b8e5a7d0ad: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.005s
I20260926 15:13:53.979972 32252 tablet_bootstrap.cc:654] T 5de5b8fb6e674922872517d886003821 P 3732c46edbc046aaad375846a0058925: Neither blocks nor log segments found. Creating new log.
I20260926 15:13:53.980846 32252 log.cc:824] T 5de5b8fb6e674922872517d886003821 P 3732c46edbc046aaad375846a0058925: Log is configured to *not* fsync() on all Append() calls
I20260926 15:13:53.981052 32251 tablet_bootstrap.cc:492] T 5de5b8fb6e674922872517d886003821 P b9d63fc120704294ad2e422ba6fd74ec: No bootstrap required, opened a new log
I20260926 15:13:53.981343 32251 ts_tablet_manager.cc:1397] T 5de5b8fb6e674922872517d886003821 P b9d63fc120704294ad2e422ba6fd74ec: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.002s
I20260926 15:13:53.984222 32252 tablet_bootstrap.cc:492] T 5de5b8fb6e674922872517d886003821 P 3732c46edbc046aaad375846a0058925: No bootstrap required, opened a new log
I20260926 15:13:53.984544 32252 ts_tablet_manager.cc:1397] T 5de5b8fb6e674922872517d886003821 P 3732c46edbc046aaad375846a0058925: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260926 15:13:53.989688 32250 raft_consensus.cc:348] T 5de5b8fb6e674922872517d886003821 P d68b0d411d1b4e68a77b56b8e5a7d0ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9d63fc120704294ad2e422ba6fd74ec" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 34237 } } peers { permanent_uuid: "3732c46edbc046aaad375846a0058925" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 38607 } } peers { permanent_uuid: "d68b0d411d1b4e68a77b56b8e5a7d0ad" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 38529 } }
I20260926 15:13:53.990166 32250 raft_consensus.cc:374] T 5de5b8fb6e674922872517d886003821 P d68b0d411d1b4e68a77b56b8e5a7d0ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:13:53.990309 32250 raft_consensus.cc:729] T 5de5b8fb6e674922872517d886003821 P d68b0d411d1b4e68a77b56b8e5a7d0ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d68b0d411d1b4e68a77b56b8e5a7d0ad, State: Initialized, Role: FOLLOWER
I20260926 15:13:53.990954 32250 consensus_queue.cc:260] T 5de5b8fb6e674922872517d886003821 P d68b0d411d1b4e68a77b56b8e5a7d0ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9d63fc120704294ad2e422ba6fd74ec" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 34237 } } peers { permanent_uuid: "3732c46edbc046aaad375846a0058925" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 38607 } } peers { permanent_uuid: "d68b0d411d1b4e68a77b56b8e5a7d0ad" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 38529 } }
I20260926 15:13:53.992080 32099 heartbeater.cc:503] Master 127.12.34.254:43809 was elected leader, sending a full tablet report...
I20260926 15:13:53.991935 32251 raft_consensus.cc:348] T 5de5b8fb6e674922872517d886003821 P b9d63fc120704294ad2e422ba6fd74ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9d63fc120704294ad2e422ba6fd74ec" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 34237 } } peers { permanent_uuid: "3732c46edbc046aaad375846a0058925" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 38607 } } peers { permanent_uuid: "d68b0d411d1b4e68a77b56b8e5a7d0ad" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 38529 } }
I20260926 15:13:53.992339 32251 raft_consensus.cc:374] T 5de5b8fb6e674922872517d886003821 P b9d63fc120704294ad2e422ba6fd74ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:13:53.992443 32251 raft_consensus.cc:729] T 5de5b8fb6e674922872517d886003821 P b9d63fc120704294ad2e422ba6fd74ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9d63fc120704294ad2e422ba6fd74ec, State: Initialized, Role: FOLLOWER
I20260926 15:13:53.992816 32250 ts_tablet_manager.cc:1428] T 5de5b8fb6e674922872517d886003821 P d68b0d411d1b4e68a77b56b8e5a7d0ad: Time spent starting tablet: real 0.014s user 0.012s sys 0.000s
I20260926 15:13:53.993153 32251 consensus_queue.cc:260] T 5de5b8fb6e674922872517d886003821 P b9d63fc120704294ad2e422ba6fd74ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9d63fc120704294ad2e422ba6fd74ec" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 34237 } } peers { permanent_uuid: "3732c46edbc046aaad375846a0058925" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 38607 } } peers { permanent_uuid: "d68b0d411d1b4e68a77b56b8e5a7d0ad" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 38529 } }
I20260926 15:13:53.991925 32252 raft_consensus.cc:348] T 5de5b8fb6e674922872517d886003821 P 3732c46edbc046aaad375846a0058925 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9d63fc120704294ad2e422ba6fd74ec" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 34237 } } peers { permanent_uuid: "3732c46edbc046aaad375846a0058925" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 38607 } } peers { permanent_uuid: "d68b0d411d1b4e68a77b56b8e5a7d0ad" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 38529 } }
I20260926 15:13:53.993471 32252 raft_consensus.cc:374] T 5de5b8fb6e674922872517d886003821 P 3732c46edbc046aaad375846a0058925 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:13:53.993578 32252 raft_consensus.cc:729] T 5de5b8fb6e674922872517d886003821 P 3732c46edbc046aaad375846a0058925 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3732c46edbc046aaad375846a0058925, State: Initialized, Role: FOLLOWER
I20260926 15:13:53.994236 32232 heartbeater.cc:503] Master 127.12.34.254:43809 was elected leader, sending a full tablet report...
I20260926 15:13:53.994320 32252 consensus_queue.cc:260] T 5de5b8fb6e674922872517d886003821 P 3732c46edbc046aaad375846a0058925 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9d63fc120704294ad2e422ba6fd74ec" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 34237 } } peers { permanent_uuid: "3732c46edbc046aaad375846a0058925" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 38607 } } peers { permanent_uuid: "d68b0d411d1b4e68a77b56b8e5a7d0ad" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 38529 } }
I20260926 15:13:53.994875 32251 ts_tablet_manager.cc:1428] T 5de5b8fb6e674922872517d886003821 P b9d63fc120704294ad2e422ba6fd74ec: Time spent starting tablet: real 0.013s user 0.012s sys 0.002s
I20260926 15:13:53.996325 32252 ts_tablet_manager.cc:1428] T 5de5b8fb6e674922872517d886003821 P 3732c46edbc046aaad375846a0058925: Time spent starting tablet: real 0.012s user 0.007s sys 0.003s
I20260926 15:13:54.027518 32257 raft_consensus.cc:482] T 5de5b8fb6e674922872517d886003821 P b9d63fc120704294ad2e422ba6fd74ec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 15:13:54.027781 32257 raft_consensus.cc:504] T 5de5b8fb6e674922872517d886003821 P b9d63fc120704294ad2e422ba6fd74ec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9d63fc120704294ad2e422ba6fd74ec" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 34237 } } peers { permanent_uuid: "3732c46edbc046aaad375846a0058925" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 38607 } } peers { permanent_uuid: "d68b0d411d1b4e68a77b56b8e5a7d0ad" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 38529 } }
I20260926 15:13:54.029651 32257 leader_election.cc:288] T 5de5b8fb6e674922872517d886003821 P b9d63fc120704294ad2e422ba6fd74ec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3732c46edbc046aaad375846a0058925 (127.12.34.193:38607), d68b0d411d1b4e68a77b56b8e5a7d0ad (127.12.34.194:38529)
I20260926 15:13:54.035701 32234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:54.029906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.194:38529 (local address 127.12.34.195:41707)
0926 15:13:54.030162 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0926 15:13:54.030178 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 15:13:54.030270 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:13:54.030875 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:13:54.030885 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:13:54.030922 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:13:54.031176 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:13:54.031187 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:13:54.032404 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:13:54.032411 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:13:54.033858 (+ 1447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:13:54.033871 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:13:54.034952 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:13:54.034958 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:13:54.035155 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:13:54.035180 (+ 25us) client_negotiation.cc:769] Sending connection context
0926 15:13:54.035288 (+ 108us) client_negotiation.cc:240] Negotiation successful
0926 15:13:54.035422 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"mutex_wait_us":20}
I20260926 15:13:54.035703 32249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:54.030288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.195:41707 (local address 127.12.34.194:38529)
0926 15:13:54.030493 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0926 15:13:54.030499 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:13:54.030519 (+ 20us) server_negotiation.cc:407] Connection header received
0926 15:13:54.030598 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:13:54.030606 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:13:54.030704 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:13:54.030837 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:13:54.031403 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:54.032258 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:54.034022 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:54.034793 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:54.034969 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:13:54.035044 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 15:13:54.035315 (+ 271us) server_negotiation.cc:299] Negotiation successful
0926 15:13:54.035435 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 15:13:54.036502 32247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:54.029998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.195:45161 (local address 127.12.34.193:38607)
0926 15:13:54.030222 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0926 15:13:54.030229 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:13:54.031303 (+ 1074us) server_negotiation.cc:407] Connection header received
0926 15:13:54.031475 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:13:54.031479 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:13:54.031537 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:13:54.031637 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:13:54.032226 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:54.033120 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:54.034889 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:54.035627 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:54.035800 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:13:54.035868 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 15:13:54.036139 (+ 271us) server_negotiation.cc:299] Negotiation successful
0926 15:13:54.036264 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 15:13:54.036580 32259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:54.029999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.193:38607 (local address 127.12.34.195:45161)
0926 15:13:54.031186 (+ 1187us) negotiation.cc:107] Waiting for socket to connect
0926 15:13:54.031204 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 15:13:54.031318 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:13:54.031655 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:13:54.031659 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:13:54.031684 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:13:54.032071 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:13:54.032081 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:13:54.033261 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:13:54.033266 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:13:54.034738 (+ 1472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:13:54.034751 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:13:54.035817 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:13:54.035823 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:13:54.035980 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:13:54.036002 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 15:13:54.036104 (+ 102us) client_negotiation.cc:240] Negotiation successful
0926 15:13:54.036217 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1065,"mutex_wait_us":409,"thread_start_us":397,"threads_started":1}
I20260926 15:13:54.037115 32054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5de5b8fb6e674922872517d886003821" candidate_uuid: "b9d63fc120704294ad2e422ba6fd74ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d68b0d411d1b4e68a77b56b8e5a7d0ad" is_pre_election: true
I20260926 15:13:54.037596 32054 raft_consensus.cc:2454] T 5de5b8fb6e674922872517d886003821 P d68b0d411d1b4e68a77b56b8e5a7d0ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d63fc120704294ad2e422ba6fd74ec in term 0.
I20260926 15:13:54.037500 31920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5de5b8fb6e674922872517d886003821" candidate_uuid: "b9d63fc120704294ad2e422ba6fd74ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3732c46edbc046aaad375846a0058925" is_pre_election: true
I20260926 15:13:54.037979 31920 raft_consensus.cc:2454] T 5de5b8fb6e674922872517d886003821 P 3732c46edbc046aaad375846a0058925 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d63fc120704294ad2e422ba6fd74ec in term 0.
I20260926 15:13:54.038266 32121 leader_election.cc:302] T 5de5b8fb6e674922872517d886003821 P b9d63fc120704294ad2e422ba6fd74ec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9d63fc120704294ad2e422ba6fd74ec, d68b0d411d1b4e68a77b56b8e5a7d0ad; no voters:
I20260926 15:13:54.038661 32257 raft_consensus.cc:2781] T 5de5b8fb6e674922872517d886003821 P b9d63fc120704294ad2e422ba6fd74ec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 15:13:54.038802 32257 raft_consensus.cc:482] T 5de5b8fb6e674922872517d886003821 P b9d63fc120704294ad2e422ba6fd74ec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 15:13:54.038902 32257 raft_consensus.cc:3037] T 5de5b8fb6e674922872517d886003821 P b9d63fc120704294ad2e422ba6fd74ec [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:13:54.041325 32257 raft_consensus.cc:504] T 5de5b8fb6e674922872517d886003821 P b9d63fc120704294ad2e422ba6fd74ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9d63fc120704294ad2e422ba6fd74ec" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 34237 } } peers { permanent_uuid: "3732c46edbc046aaad375846a0058925" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 38607 } } peers { permanent_uuid: "d68b0d411d1b4e68a77b56b8e5a7d0ad" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 38529 } }
I20260926 15:13:54.042007 32257 leader_election.cc:288] T 5de5b8fb6e674922872517d886003821 P b9d63fc120704294ad2e422ba6fd74ec [CANDIDATE]: Term 1 election: Requested vote from peers 3732c46edbc046aaad375846a0058925 (127.12.34.193:38607), d68b0d411d1b4e68a77b56b8e5a7d0ad (127.12.34.194:38529)
I20260926 15:13:54.042414 31920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5de5b8fb6e674922872517d886003821" candidate_uuid: "b9d63fc120704294ad2e422ba6fd74ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3732c46edbc046aaad375846a0058925"
I20260926 15:13:54.042459 32054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5de5b8fb6e674922872517d886003821" candidate_uuid: "b9d63fc120704294ad2e422ba6fd74ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d68b0d411d1b4e68a77b56b8e5a7d0ad"
I20260926 15:13:54.042604 31920 raft_consensus.cc:3037] T 5de5b8fb6e674922872517d886003821 P 3732c46edbc046aaad375846a0058925 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:13:54.042721 32054 raft_consensus.cc:3037] T 5de5b8fb6e674922872517d886003821 P d68b0d411d1b4e68a77b56b8e5a7d0ad [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:13:54.044790 31920 raft_consensus.cc:2454] T 5de5b8fb6e674922872517d886003821 P 3732c46edbc046aaad375846a0058925 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9d63fc120704294ad2e422ba6fd74ec in term 1.
I20260926 15:13:54.044926 32054 raft_consensus.cc:2454] T 5de5b8fb6e674922872517d886003821 P d68b0d411d1b4e68a77b56b8e5a7d0ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9d63fc120704294ad2e422ba6fd74ec in term 1.
I20260926 15:13:54.045226 32120 leader_election.cc:302] T 5de5b8fb6e674922872517d886003821 P b9d63fc120704294ad2e422ba6fd74ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3732c46edbc046aaad375846a0058925, b9d63fc120704294ad2e422ba6fd74ec; no voters:
I20260926 15:13:54.045522 32257 raft_consensus.cc:2781] T 5de5b8fb6e674922872517d886003821 P b9d63fc120704294ad2e422ba6fd74ec [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:13:54.045957 32257 raft_consensus.cc:686] T 5de5b8fb6e674922872517d886003821 P b9d63fc120704294ad2e422ba6fd74ec [term 1 LEADER]: Becoming Leader. State: Replica: b9d63fc120704294ad2e422ba6fd74ec, State: Running, Role: LEADER
I20260926 15:13:54.046422 32257 consensus_queue.cc:237] T 5de5b8fb6e674922872517d886003821 P b9d63fc120704294ad2e422ba6fd74ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9d63fc120704294ad2e422ba6fd74ec" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 34237 } } peers { permanent_uuid: "3732c46edbc046aaad375846a0058925" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 38607 } } peers { permanent_uuid: "d68b0d411d1b4e68a77b56b8e5a7d0ad" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 38529 } }
I20260926 15:13:54.050742 31776 catalog_manager.cc:5495] T 5de5b8fb6e674922872517d886003821 P b9d63fc120704294ad2e422ba6fd74ec reported cstate change: term changed from 0 to 1, leader changed from <none> to b9d63fc120704294ad2e422ba6fd74ec (127.12.34.195). New cstate: current_term: 1 leader_uuid: "b9d63fc120704294ad2e422ba6fd74ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9d63fc120704294ad2e422ba6fd74ec" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 34237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3732c46edbc046aaad375846a0058925" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 38607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d68b0d411d1b4e68a77b56b8e5a7d0ad" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 38529 } health_report { overall_health: UNKNOWN } } }
I20260926 15:13:54.077425 32246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:54.071970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36618 (local address 127.12.34.195:34237)
0926 15:13:54.072181 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0926 15:13:54.072189 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 15:13:54.072208 (+ 19us) server_negotiation.cc:407] Connection header received
0926 15:13:54.072479 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:13:54.072490 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:13:54.072570 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:13:54.072733 (+ 163us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 15:13:54.073127 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:54.074013 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:54.075385 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:54.075773 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:54.075903 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:13:54.076369 (+ 466us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 15:13:54.076841 (+ 472us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 15:13:54.077005 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0926 15:13:54.077069 (+ 64us) server_negotiation.cc:299] Negotiation successful
0926 15:13:54.077167 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 15:13:54.085599 31920 raft_consensus.cc:1261] T 5de5b8fb6e674922872517d886003821 P 3732c46edbc046aaad375846a0058925 [term 1 FOLLOWER]: Refusing update from remote peer b9d63fc120704294ad2e422ba6fd74ec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:13:54.085614 32054 raft_consensus.cc:1261] T 5de5b8fb6e674922872517d886003821 P d68b0d411d1b4e68a77b56b8e5a7d0ad [term 1 FOLLOWER]: Refusing update from remote peer b9d63fc120704294ad2e422ba6fd74ec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:13:54.086511 32260 consensus_queue.cc:1035] T 5de5b8fb6e674922872517d886003821 P b9d63fc120704294ad2e422ba6fd74ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "d68b0d411d1b4e68a77b56b8e5a7d0ad" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 38529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:13:54.086916 32257 consensus_queue.cc:1035] T 5de5b8fb6e674922872517d886003821 P b9d63fc120704294ad2e422ba6fd74ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "3732c46edbc046aaad375846a0058925" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 38607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:13:54.098843 32263 mvcc.cc:205] Tried to move back new op lower bound from 7333624357204508672 to 7333624357057970176. Current Snapshot: MvccSnapshot[applied={T|T < 7333624357204508672}]
I20260926 15:13:54.099315 32265 mvcc.cc:205] Tried to move back new op lower bound from 7333624357204508672 to 7333624357057970176. Current Snapshot: MvccSnapshot[applied={T|T < 7333624357204508672}]
I20260926 15:13:54.116781 31775 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60972:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 15:13:54.117111 31775 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60972:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 15:13:54.120702 31775 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d91b3e5a843e42668bd8b3f545705821: Sending DeleteTablet for 3 replicas of tablet 5de5b8fb6e674922872517d886003821
I20260926 15:13:54.121932 32167 tablet_service.cc:1514] Processing DeleteTablet for tablet 5de5b8fb6e674922872517d886003821 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:13:54 UTC) from {username='kudu', principal='kudu/127.12.34.254@KRBTEST.COM'} at 127.0.0.1:36614
I20260926 15:13:54.121956 31900 tablet_service.cc:1514] Processing DeleteTablet for tablet 5de5b8fb6e674922872517d886003821 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:13:54 UTC) from {username='kudu', principal='kudu/127.12.34.254@KRBTEST.COM'} at 127.0.0.1:34522
I20260926 15:13:54.122897 32034 tablet_service.cc:1514] Processing DeleteTablet for tablet 5de5b8fb6e674922872517d886003821 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:13:54 UTC) from {username='kudu', principal='kudu/127.12.34.254@KRBTEST.COM'} at 127.0.0.1:58338
I20260926 15:13:54.123180 32274 tablet_replica.cc:330] T 5de5b8fb6e674922872517d886003821 P 3732c46edbc046aaad375846a0058925: stopping tablet replica
I20260926 15:13:54.123571 32274 raft_consensus.cc:2229] T 5de5b8fb6e674922872517d886003821 P 3732c46edbc046aaad375846a0058925 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 15:13:54.124061 32274 raft_consensus.cc:2258] T 5de5b8fb6e674922872517d886003821 P 3732c46edbc046aaad375846a0058925 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:13:54.125276 12427 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 31837
I20260926 15:13:54.134676 32274 ts_tablet_manager.cc:1905] T 5de5b8fb6e674922872517d886003821 P 3732c46edbc046aaad375846a0058925: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:13:54.142590 32275 tablet_replica.cc:330] T 5de5b8fb6e674922872517d886003821 P d68b0d411d1b4e68a77b56b8e5a7d0ad: stopping tablet replica
I20260926 15:13:54.143297 32275 raft_consensus.cc:2229] T 5de5b8fb6e674922872517d886003821 P d68b0d411d1b4e68a77b56b8e5a7d0ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 15:13:54.143990 32275 raft_consensus.cc:2258] T 5de5b8fb6e674922872517d886003821 P d68b0d411d1b4e68a77b56b8e5a7d0ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:13:54.146572 32275 ts_tablet_manager.cc:1905] T 5de5b8fb6e674922872517d886003821 P d68b0d411d1b4e68a77b56b8e5a7d0ad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:13:54.147362 32273 tablet_replica.cc:330] T 5de5b8fb6e674922872517d886003821 P b9d63fc120704294ad2e422ba6fd74ec: stopping tablet replica
I20260926 15:13:54.147853 32273 raft_consensus.cc:2229] T 5de5b8fb6e674922872517d886003821 P b9d63fc120704294ad2e422ba6fd74ec [term 1 LEADER]: Raft consensus shutting down.
I20260926 15:13:54.148729 32273 raft_consensus.cc:2258] T 5de5b8fb6e674922872517d886003821 P b9d63fc120704294ad2e422ba6fd74ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:13:54.156498 32275 ts_tablet_manager.cc:1918] T 5de5b8fb6e674922872517d886003821 P d68b0d411d1b4e68a77b56b8e5a7d0ad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 15:13:54.156745 32275 log.cc:1196] T 5de5b8fb6e674922872517d886003821 P d68b0d411d1b4e68a77b56b8e5a7d0ad: Deleting WAL directory at /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435318278681-12427-0/minicluster-data/ts-1/wal/wals/5de5b8fb6e674922872517d886003821
I20260926 15:13:54.157413 32275 ts_tablet_manager.cc:1939] T 5de5b8fb6e674922872517d886003821 P d68b0d411d1b4e68a77b56b8e5a7d0ad: Deleting consensus metadata
I20260926 15:13:54.158107 32273 ts_tablet_manager.cc:1905] T 5de5b8fb6e674922872517d886003821 P b9d63fc120704294ad2e422ba6fd74ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:13:54.159402 31761 catalog_manager.cc:4841] TS d68b0d411d1b4e68a77b56b8e5a7d0ad (127.12.34.194:38529): tablet 5de5b8fb6e674922872517d886003821 (table test-table [id=ec81e74d4f14494bab66287e2119f6c4]) successfully deleted
Sep 26 15:13:54 dist-test-slave-865v krb5kdc[31731](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435631, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.34.193@KRBTEST.COM
I20260926 15:13:54.161243 32274 ts_tablet_manager.cc:1918] T 5de5b8fb6e674922872517d886003821 P 3732c46edbc046aaad375846a0058925: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 15:13:54.161520 32274 log.cc:1196] T 5de5b8fb6e674922872517d886003821 P 3732c46edbc046aaad375846a0058925: Deleting WAL directory at /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435318278681-12427-0/minicluster-data/ts-0/wal/wals/5de5b8fb6e674922872517d886003821
I20260926 15:13:54.162091 32274 ts_tablet_manager.cc:1939] T 5de5b8fb6e674922872517d886003821 P 3732c46edbc046aaad375846a0058925: Deleting consensus metadata
I20260926 15:13:54.164971 31760 catalog_manager.cc:4841] TS 3732c46edbc046aaad375846a0058925 (127.12.34.193:38607): tablet 5de5b8fb6e674922872517d886003821 (table test-table [id=ec81e74d4f14494bab66287e2119f6c4]) successfully deleted
I20260926 15:13:54.166136 32273 ts_tablet_manager.cc:1918] T 5de5b8fb6e674922872517d886003821 P b9d63fc120704294ad2e422ba6fd74ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 15:13:54.166309 32273 log.cc:1196] T 5de5b8fb6e674922872517d886003821 P b9d63fc120704294ad2e422ba6fd74ec: Deleting WAL directory at /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790435318278681-12427-0/minicluster-data/ts-2/wal/wals/5de5b8fb6e674922872517d886003821
I20260926 15:13:54.166940 32273 ts_tablet_manager.cc:1939] T 5de5b8fb6e674922872517d886003821 P b9d63fc120704294ad2e422ba6fd74ec: Deleting consensus metadata
I20260926 15:13:54.167833 32247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:54.126234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34536 (local address 127.12.34.193:38607)
0926 15:13:54.126432 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0926 15:13:54.126438 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:13:54.149538 (+ 23100us) server_negotiation.cc:407] Connection header received
0926 15:13:54.149745 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:13:54.149750 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:13:54.149839 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:13:54.149946 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:13:54.151598 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:54.152712 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:54.154600 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:54.155235 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:54.155354 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:13:54.160832 (+ 5478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:13:54.160843 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:13:54.160856 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:13:54.160890 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:13:54.164591 (+ 3701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:13:54.165430 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:13:54.165445 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:13:54.165447 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:13:54.165570 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:13:54.165938 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:13:54.165943 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:13:54.165946 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:13:54.166328 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:13:54.166638 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0926 15:13:54.167096 (+ 458us) server_negotiation.cc:299] Negotiation successful
0926 15:13:54.167314 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 15:13:54.168495 31960 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:13:54.168568 31762 catalog_manager.cc:4841] TS b9d63fc120704294ad2e422ba6fd74ec (127.12.34.195:34237): tablet 5de5b8fb6e674922872517d886003821 (table test-table [id=ec81e74d4f14494bab66287e2119f6c4]) successfully deleted
I20260926 15:13:54.259522 12427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8uoJGv/build/asan/bin/kudu with pid 31837
I20260926 15:13:54.277316 12427 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 31971
Sep 26 15:13:54 dist-test-slave-865v krb5kdc[31731](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435631, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.34.194@KRBTEST.COM
I20260926 15:13:54.294051 32249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:54.278077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58340 (local address 127.12.34.194:38529)
0926 15:13:54.278239 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0926 15:13:54.278244 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:13:54.278262 (+ 18us) server_negotiation.cc:407] Connection header received
0926 15:13:54.278506 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:13:54.278510 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:13:54.278578 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:13:54.278667 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:13:54.280294 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:54.281044 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:54.283086 (+ 2042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:54.283477 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:54.283573 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:13:54.288467 (+ 4894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:13:54.288479 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:13:54.288494 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:13:54.288526 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:13:54.291298 (+ 2772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:13:54.292000 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:13:54.292007 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:13:54.292009 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:13:54.292125 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:13:54.292498 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:13:54.292504 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:13:54.292507 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:13:54.292882 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:13:54.293195 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0926 15:13:54.293501 (+ 306us) server_negotiation.cc:299] Negotiation successful
0926 15:13:54.293713 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 15:13:54.294667 32094 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:13:54.383848 12427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8uoJGv/build/asan/bin/kudu with pid 31971
I20260926 15:13:54.401690 12427 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 32104
Sep 26 15:13:54 dist-test-slave-865v krb5kdc[31731](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435631, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.34.195@KRBTEST.COM
I20260926 15:13:54.417850 32246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:54.402499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36634 (local address 127.12.34.195:34237)
0926 15:13:54.402674 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0926 15:13:54.402683 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 15:13:54.402707 (+ 24us) server_negotiation.cc:407] Connection header received
0926 15:13:54.402928 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:13:54.402934 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:13:54.403008 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:13:54.403141 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:13:54.404340 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:54.405342 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:54.406666 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:54.407127 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:54.407275 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:13:54.411559 (+ 4284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:13:54.411570 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:13:54.411585 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:13:54.411621 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:13:54.415019 (+ 3398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:13:54.415698 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:13:54.415705 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:13:54.415707 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:13:54.415870 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:13:54.416240 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:13:54.416245 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:13:54.416250 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:13:54.416610 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:13:54.416953 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0926 15:13:54.417264 (+ 311us) server_negotiation.cc:299] Negotiation successful
0926 15:13:54.417496 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 15:13:54.418532 32227 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:13:54.539358 12427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8uoJGv/build/asan/bin/kudu with pid 32104
I20260926 15:13:54.558637 12427 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 31744
I20260926 15:13:54.559687 31806 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:13:54.674685 12427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8uoJGv/build/asan/bin/kudu with pid 31744
2026-09-26T15:13:54Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4297 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-26T15:13:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T15:13:54Z Disabled control of system clock
I20260926 15:13:54.751588 12427 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8uoJGv/build/asan/bin/kudu
/tmp/dist-test-task8uoJGv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790435318278681-12427-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790435318278681-12427-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790435318278681-12427-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.34.254:38865
--webserver_interface=127.12.34.254
--webserver_port=0
--builtin_ntp_servers=127.12.34.212:38985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.34.254:38865
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 15:13:55.083032 32284 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:13:55.083333 32284 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:13:55.083422 32284 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:13:55.091522 32284 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:13:55.091646 32284 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:13:55.091702 32284 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:13:55.091758 32284 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:13:55.091809 32284 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:13:55.101802 32284 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.34.212:38985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790435318278681-12427-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790435318278681-12427-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.34.254:38865
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.34.254:38865
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.34.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790435318278681-12427-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:13:55.103102 32284 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:13:55.104853 32284 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:13:55.117235 32289 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:13:55.117264 32290 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:13:55.117472 32292 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:13:55.118099 32284 server_base.cc:1034] running on GCE node
I20260926 15:13:55.118921 32284 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:13:55.120235 32284 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:13:55.121438 32284 hybrid_clock.cc:648] HybridClock initialized: now 1790435635121374 us; error 51 us; skew 500 ppm
I20260926 15:13:55.121837 32284 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 15:13:55.124104 32284 webserver.cc:460] Webserver started at http://127.12.34.254:42183/ using document root <none> and password file <none>
I20260926 15:13:55.124769 32284 fs_manager.cc:359] Metadata directory not provided
I20260926 15:13:55.124883 32284 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:13:55.125267 32284 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:13:55.128059 32284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790435318278681-12427-0/minicluster-data/master-0/data/instance:
uuid: "8522f440d0e240dbac8cf905b2190263"
format_stamp: "Formatted at 2026-09-26 15:13:55 on dist-test-slave-865v"
server_key: "4b501b97a124d5a54ee59c2b57cafe4e"
server_key_iv: "0d7f683336fdb20c359bff8caa8bba37"
server_key_version: "encryptionkey@0"
I20260926 15:13:55.128842 32284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790435318278681-12427-0/minicluster-data/master-0/wal/instance:
uuid: "8522f440d0e240dbac8cf905b2190263"
format_stamp: "Formatted at 2026-09-26 15:13:55 on dist-test-slave-865v"
server_key: "4b501b97a124d5a54ee59c2b57cafe4e"
server_key_iv: "0d7f683336fdb20c359bff8caa8bba37"
server_key_version: "encryptionkey@0"
I20260926 15:13:55.133941 32284 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260926 15:13:55.137470 32298 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:13:55.138831 32284 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 15:13:55.139005 32284 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790435318278681-12427-0/minicluster-data/master-0/data,/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790435318278681-12427-0/minicluster-data/master-0/wal
uuid: "8522f440d0e240dbac8cf905b2190263"
format_stamp: "Formatted at 2026-09-26 15:13:55 on dist-test-slave-865v"
server_key: "4b501b97a124d5a54ee59c2b57cafe4e"
server_key_iv: "0d7f683336fdb20c359bff8caa8bba37"
server_key_version: "encryptionkey@0"
I20260926 15:13:55.139235 32284 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790435318278681-12427-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790435318278681-12427-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790435318278681-12427-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:13:55.160328 32284 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:13:55.161110 32284 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:13:55.161394 32284 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:13:55.179299 32284 rpc_server.cc:307] RPC server started. Bound to: 127.12.34.254:38865
I20260926 15:13:55.179323 32349 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.34.254:38865 every 8 connection(s)
I20260926 15:13:55.180536 32284 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
I20260926 15:13:55.184163 32350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:13:55.184367 12427 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8uoJGv/build/asan/bin/kudu as pid 32284
I20260926 15:13:55.184527 12427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790435318278681-12427-0/minicluster-data/master-0/wal/instance
I20260926 15:13:55.185127 12427 external_mini_cluster.cc:1428] Setting key 617a31bd8b0eff8f64cfb6017de0d464
I20260926 15:13:55.196359 32350 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8522f440d0e240dbac8cf905b2190263: Bootstrap starting.
I20260926 15:13:55.198530 32353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:55.187017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41856 (local address 127.12.34.254:38865)
0926 15:13:55.188003 (+ 986us) server_negotiation.cc:206] Beginning negotiation
0926 15:13:55.188013 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 15:13:55.188052 (+ 39us) server_negotiation.cc:407] Connection header received
0926 15:13:55.190374 (+ 2322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:13:55.190392 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:13:55.190821 (+ 429us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:13:55.191198 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:13:55.191713 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:55.192892 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:55.194291 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:55.194995 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:55.195188 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:13:55.196350 (+ 1162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:13:55.196358 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:13:55.196369 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:13:55.196912 (+ 543us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:13:55.196983 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:13:55.197003 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:13:55.197143 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0926 15:13:55.197359 (+ 216us) server_negotiation.cc:299] Negotiation successful
0926 15:13:55.197632 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":589,"thread_start_us":357,"threads_started":1}
I20260926 15:13:55.199914 32350 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8522f440d0e240dbac8cf905b2190263: Neither blocks nor log segments found. Creating new log.
I20260926 15:13:55.200911 32350 log.cc:824] T 00000000000000000000000000000000 P 8522f440d0e240dbac8cf905b2190263: Log is configured to *not* fsync() on all Append() calls
I20260926 15:13:55.203603 32350 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8522f440d0e240dbac8cf905b2190263: No bootstrap required, opened a new log
I20260926 15:13:55.211124 32350 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8522f440d0e240dbac8cf905b2190263 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8522f440d0e240dbac8cf905b2190263" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 38865 } }
I20260926 15:13:55.211562 32350 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8522f440d0e240dbac8cf905b2190263 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:13:55.211683 32350 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8522f440d0e240dbac8cf905b2190263 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8522f440d0e240dbac8cf905b2190263, State: Initialized, Role: FOLLOWER
I20260926 15:13:55.212231 32350 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8522f440d0e240dbac8cf905b2190263 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8522f440d0e240dbac8cf905b2190263" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 38865 } }
I20260926 15:13:55.212471 32350 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8522f440d0e240dbac8cf905b2190263 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 15:13:55.212625 32350 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8522f440d0e240dbac8cf905b2190263 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 15:13:55.212800 32350 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8522f440d0e240dbac8cf905b2190263 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:13:55.214936 32350 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8522f440d0e240dbac8cf905b2190263 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8522f440d0e240dbac8cf905b2190263" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 38865 } }
I20260926 15:13:55.215412 32350 leader_election.cc:302] T 00000000000000000000000000000000 P 8522f440d0e240dbac8cf905b2190263 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8522f440d0e240dbac8cf905b2190263; no voters:
I20260926 15:13:55.216033 32350 leader_election.cc:288] T 00000000000000000000000000000000 P 8522f440d0e240dbac8cf905b2190263 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 15:13:55.216202 32355 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8522f440d0e240dbac8cf905b2190263 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:13:55.216972 32355 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8522f440d0e240dbac8cf905b2190263 [term 1 LEADER]: Becoming Leader. State: Replica: 8522f440d0e240dbac8cf905b2190263, State: Running, Role: LEADER
I20260926 15:13:55.217527 32355 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8522f440d0e240dbac8cf905b2190263 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8522f440d0e240dbac8cf905b2190263" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 38865 } }
I20260926 15:13:55.218176 32350 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8522f440d0e240dbac8cf905b2190263 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:13:55.220218 32357 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8522f440d0e240dbac8cf905b2190263 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8522f440d0e240dbac8cf905b2190263. Latest consensus state: current_term: 1 leader_uuid: "8522f440d0e240dbac8cf905b2190263" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8522f440d0e240dbac8cf905b2190263" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 38865 } } }
I20260926 15:13:55.220583 32357 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8522f440d0e240dbac8cf905b2190263 [sys.catalog]: This master's current role is: LEADER
I20260926 15:13:55.221406 32356 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8522f440d0e240dbac8cf905b2190263 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8522f440d0e240dbac8cf905b2190263" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8522f440d0e240dbac8cf905b2190263" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 38865 } } }
I20260926 15:13:55.221803 32356 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8522f440d0e240dbac8cf905b2190263 [sys.catalog]: This master's current role is: LEADER
I20260926 15:13:55.222427 32360 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:13:55.228240 32360 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 15:13:55.228952 32372 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 15:13:55.237156 32360 catalog_manager.cc:1347] Generated new cluster ID: ee61c937d46b4929851a6b760ddfaa25
I20260926 15:13:55.237272 32360 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 15:13:55.256577 32360 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 15:13:55.257930 32360 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:13:55.268288 32360 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8522f440d0e240dbac8cf905b2190263: Generated new TSK 0
I20260926 15:13:55.269136 32360 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 15:13:55.284366 12427 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8uoJGv/build/asan/bin/kudu
/tmp/dist-test-task8uoJGv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790435318278681-12427-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790435318278681-12427-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790435318278681-12427-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790435318278681-12427-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.34.193:0
--local_ip_for_outbound_sockets=127.12.34.193
--webserver_interface=127.12.34.193
--webserver_port=0
--tserver_master_addrs=127.12.34.254:38865
--builtin_ntp_servers=127.12.34.212:38985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 15:13:55.610525 32374 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:13:55.610831 32374 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:13:55.610986 32374 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:13:55.618281 32374 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:13:55.618402 32374 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:13:55.618568 32374 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.34.193
I20260926 15:13:55.627657 32374 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.34.212:38985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790435318278681-12427-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790435318278681-12427-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.34.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790435318278681-12427-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.34.193
--webserver_port=0
--tserver_master_addrs=127.12.34.254:38865
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.34.193
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790435318278681-12427-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:13:55.628863 32374 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:13:55.630558 32374 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:13:55.643448 32380 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:13:55.643620 32379 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:13:55.643500 32382 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:13:55.644757 32374 server_base.cc:1034] running on GCE node
I20260926 15:13:55.645496 32374 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:13:55.646423 32374 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:13:55.647647 32374 hybrid_clock.cc:648] HybridClock initialized: now 1790435635647594 us; error 40 us; skew 500 ppm
I20260926 15:13:55.648018 32374 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 15:13:55.650391 32374 webserver.cc:460] Webserver started at http://127.12.34.193:42329/ using document root <none> and password file <none>
I20260926 15:13:55.651139 32374 fs_manager.cc:359] Metadata directory not provided
I20260926 15:13:55.651238 32374 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:13:55.651595 32374 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:13:55.654630 32374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790435318278681-12427-0/minicluster-data/ts-0/data/instance:
uuid: "d5f58c067f4348428a28b86bed674028"
format_stamp: "Formatted at 2026-09-26 15:13:55 on dist-test-slave-865v"
server_key: "8ee41c9039782dd890a491b523fa577b"
server_key_iv: "5409a509d3a6a22f213e763d8bf6e245"
server_key_version: "encryptionkey@0"
I20260926 15:13:55.655455 32374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790435318278681-12427-0/minicluster-data/ts-0/wal/instance:
uuid: "d5f58c067f4348428a28b86bed674028"
format_stamp: "Formatted at 2026-09-26 15:13:55 on dist-test-slave-865v"
server_key: "8ee41c9039782dd890a491b523fa577b"
server_key_iv: "5409a509d3a6a22f213e763d8bf6e245"
server_key_version: "encryptionkey@0"
I20260926 15:13:55.660857 32374 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260926 15:13:55.664461 32388 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:13:55.666160 32374 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 15:13:55.666348 32374 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790435318278681-12427-0/minicluster-data/ts-0/data,/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790435318278681-12427-0/minicluster-data/ts-0/wal
uuid: "d5f58c067f4348428a28b86bed674028"
format_stamp: "Formatted at 2026-09-26 15:13:55 on dist-test-slave-865v"
server_key: "8ee41c9039782dd890a491b523fa577b"
server_key_iv: "5409a509d3a6a22f213e763d8bf6e245"
server_key_version: "encryptionkey@0"
I20260926 15:13:55.666572 32374 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790435318278681-12427-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790435318278681-12427-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790435318278681-12427-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:13:55.683856 32374 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:13:55.684636 32374 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:13:55.684938 32374 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:13:55.685920 32374 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:13:55.687332 32374 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:13:55.687455 32374 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:13:55.687588 32374 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:13:55.687681 32374 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:13:55.722406 32374 rpc_server.cc:307] RPC server started. Bound to: 127.12.34.193:40709
I20260926 15:13:55.722455 32500 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.34.193:40709 every 8 connection(s)
I20260926 15:13:55.723730 32374 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790435318278681-12427-0/minicluster-data/ts-0/data/info.pb
I20260926 15:13:55.730950 12427 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8uoJGv/build/asan/bin/kudu as pid 32374
I20260926 15:13:55.731098 12427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790435318278681-12427-0/minicluster-data/ts-0/wal/instance
I20260926 15:13:55.731531 12427 external_mini_cluster.cc:1428] Setting key a4ce36ba135207f2ba8ebb9f09d07d51
I20260926 15:13:55.735004 12427 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8uoJGv/build/asan/bin/kudu
/tmp/dist-test-task8uoJGv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790435318278681-12427-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790435318278681-12427-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790435318278681-12427-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790435318278681-12427-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.34.194:0
--local_ip_for_outbound_sockets=127.12.34.194
--webserver_interface=127.12.34.194
--webserver_port=0
--tserver_master_addrs=127.12.34.254:38865
--builtin_ntp_servers=127.12.34.212:38985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 15:13:55.738806 32503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:55.726390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.193:33515 (local address 127.12.34.254:38865)
0926 15:13:55.727156 (+ 766us) server_negotiation.cc:206] Beginning negotiation
0926 15:13:55.727166 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 15:13:55.728247 (+ 1081us) server_negotiation.cc:407] Connection header received
0926 15:13:55.730290 (+ 2043us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:13:55.730296 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:13:55.730400 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:13:55.730535 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:13:55.732461 (+ 1926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:55.733370 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:55.735475 (+ 2105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:55.736298 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:55.736459 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:13:55.737836 (+ 1377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:13:55.737852 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:13:55.737859 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:13:55.737887 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:13:55.737938 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:13:55.737956 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:13:55.738050 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 15:13:55.738272 (+ 222us) server_negotiation.cc:299] Negotiation successful
0926 15:13:55.738386 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":574,"thread_start_us":460,"threads_started":1}
I20260926 15:13:55.739655 32504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:55.726870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:38865 (local address 127.12.34.193:33515)
0926 15:13:55.728049 (+ 1179us) negotiation.cc:107] Waiting for socket to connect
0926 15:13:55.728089 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 15:13:55.729997 (+ 1908us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:13:55.730808 (+ 811us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:13:55.730817 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:13:55.730904 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:13:55.732201 (+ 1297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:13:55.732243 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:13:55.733590 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:13:55.733597 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:13:55.735313 (+ 1716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:13:55.735328 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:13:55.736531 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:13:55.736536 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:13:55.736738 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:13:55.737507 (+ 769us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 15:13:55.737510 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:13:55.737548 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 15:13:55.737625 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:13:55.738060 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:13:55.738068 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:13:55.738075 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 15:13:55.738228 (+ 153us) client_negotiation.cc:240] Negotiation successful
0926 15:13:55.738532 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":563,"thread_start_us":347,"threads_started":1}
I20260926 15:13:55.741273 32501 heartbeater.cc:347] Connected to a master server at 127.12.34.254:38865
I20260926 15:13:55.741569 32501 heartbeater.cc:464] Registering TS with master...
I20260926 15:13:55.742313 32501 heartbeater.cc:511] Master 127.12.34.254:38865 requested a full tablet report, sending...
I20260926 15:13:55.744652 32315 ts_manager.cc:194] Registered new tserver with Master: d5f58c067f4348428a28b86bed674028 (127.12.34.193:40709)
I20260926 15:13:55.747100 32315 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.34.193:33515
W20260926 15:13:56.059026 32505 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:13:56.059297 32505 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:13:56.059382 32505 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:13:56.066520 32505 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:13:56.066641 32505 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:13:56.066751 32505 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.34.194
I20260926 15:13:56.076099 32505 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.34.212:38985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790435318278681-12427-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790435318278681-12427-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.34.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790435318278681-12427-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.34.194
--webserver_port=0
--tserver_master_addrs=127.12.34.254:38865
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.34.194
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790435318278681-12427-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:13:56.077368 32505 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:13:56.078974 32505 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:13:56.091939 32510 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:13:56.092001 32513 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:13:56.091928 32511 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:13:56.092613 32505 server_base.cc:1034] running on GCE node
I20260926 15:13:56.093343 32505 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:13:56.094297 32505 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:13:56.095574 32505 hybrid_clock.cc:648] HybridClock initialized: now 1790435636095510 us; error 51 us; skew 500 ppm
I20260926 15:13:56.095935 32505 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 15:13:56.098305 32505 webserver.cc:460] Webserver started at http://127.12.34.194:39153/ using document root <none> and password file <none>
I20260926 15:13:56.098963 32505 fs_manager.cc:359] Metadata directory not provided
I20260926 15:13:56.099092 32505 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:13:56.099393 32505 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:13:56.102174 32505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790435318278681-12427-0/minicluster-data/ts-1/data/instance:
uuid: "1149b5eb4e3b4e79aa969ee23a4f09b8"
format_stamp: "Formatted at 2026-09-26 15:13:56 on dist-test-slave-865v"
server_key: "ac3f893332fcd8c20f7f6ec52b1365d0"
server_key_iv: "48af74df8ddb165cfcb9c4d95bf12326"
server_key_version: "encryptionkey@0"
I20260926 15:13:56.102919 32505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790435318278681-12427-0/minicluster-data/ts-1/wal/instance:
uuid: "1149b5eb4e3b4e79aa969ee23a4f09b8"
format_stamp: "Formatted at 2026-09-26 15:13:56 on dist-test-slave-865v"
server_key: "ac3f893332fcd8c20f7f6ec52b1365d0"
server_key_iv: "48af74df8ddb165cfcb9c4d95bf12326"
server_key_version: "encryptionkey@0"
I20260926 15:13:56.107594 32505 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260926 15:13:56.110759 32519 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:13:56.111950 32505 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260926 15:13:56.112134 32505 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790435318278681-12427-0/minicluster-data/ts-1/data,/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790435318278681-12427-0/minicluster-data/ts-1/wal
uuid: "1149b5eb4e3b4e79aa969ee23a4f09b8"
format_stamp: "Formatted at 2026-09-26 15:13:56 on dist-test-slave-865v"
server_key: "ac3f893332fcd8c20f7f6ec52b1365d0"
server_key_iv: "48af74df8ddb165cfcb9c4d95bf12326"
server_key_version: "encryptionkey@0"
I20260926 15:13:56.112362 32505 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790435318278681-12427-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790435318278681-12427-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790435318278681-12427-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:13:56.127434 32505 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:13:56.128149 32505 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:13:56.128399 32505 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:13:56.129315 32505 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:13:56.130708 32505 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:13:56.130797 32505 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:13:56.130874 32505 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:13:56.130931 32505 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:13:56.167809 32505 rpc_server.cc:307] RPC server started. Bound to: 127.12.34.194:33689
I20260926 15:13:56.167896 32631 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.34.194:33689 every 8 connection(s)
I20260926 15:13:56.169138 32505 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790435318278681-12427-0/minicluster-data/ts-1/data/info.pb
I20260926 15:13:56.172474 12427 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8uoJGv/build/asan/bin/kudu as pid 32505
I20260926 15:13:56.172650 12427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790435318278681-12427-0/minicluster-data/ts-1/wal/instance
I20260926 15:13:56.173162 12427 external_mini_cluster.cc:1428] Setting key 8615a31918d6f2e8255544ef01394ffa
I20260926 15:13:56.176219 12427 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8uoJGv/build/asan/bin/kudu
/tmp/dist-test-task8uoJGv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790435318278681-12427-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790435318278681-12427-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790435318278681-12427-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790435318278681-12427-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.34.195:0
--local_ip_for_outbound_sockets=127.12.34.195
--webserver_interface=127.12.34.195
--webserver_port=0
--tserver_master_addrs=127.12.34.254:38865
--builtin_ntp_servers=127.12.34.212:38985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 15:13:56.184090 32503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:56.171750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.194:49491 (local address 127.12.34.254:38865)
0926 15:13:56.171939 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0926 15:13:56.171944 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:13:56.173456 (+ 1512us) server_negotiation.cc:407] Connection header received
0926 15:13:56.175650 (+ 2194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:13:56.175655 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:13:56.175735 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:13:56.175848 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:13:56.177005 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:56.178201 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:56.180650 (+ 2449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:56.181460 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:56.181601 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:13:56.183124 (+ 1523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:13:56.183133 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:13:56.183141 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:13:56.183166 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:13:56.183208 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:13:56.183226 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:13:56.183321 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 15:13:56.183567 (+ 246us) server_negotiation.cc:299] Negotiation successful
0926 15:13:56.183683 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 15:13:56.185310 32634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:56.172257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:38865 (local address 127.12.34.194:49491)
0926 15:13:56.173303 (+ 1046us) negotiation.cc:107] Waiting for socket to connect
0926 15:13:56.173345 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 15:13:56.175395 (+ 2050us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:13:56.176052 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:13:56.176059 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:13:56.176140 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:13:56.176820 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:13:56.176838 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:13:56.178367 (+ 1529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:13:56.178370 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:13:56.180478 (+ 2108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:13:56.180494 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:13:56.181680 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:13:56.181687 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:13:56.181962 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:13:56.182778 (+ 816us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 15:13:56.182782 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:13:56.182828 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 15:13:56.182918 (+ 90us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:13:56.183334 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:13:56.183344 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:13:56.183352 (+ 8us) client_negotiation.cc:769] Sending connection context
0926 15:13:56.183527 (+ 175us) client_negotiation.cc:240] Negotiation successful
0926 15:13:56.183879 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":340,"threads_started":1}
I20260926 15:13:56.186734 32632 heartbeater.cc:347] Connected to a master server at 127.12.34.254:38865
I20260926 15:13:56.187042 32632 heartbeater.cc:464] Registering TS with master...
I20260926 15:13:56.187925 32632 heartbeater.cc:511] Master 127.12.34.254:38865 requested a full tablet report, sending...
I20260926 15:13:56.189483 32315 ts_manager.cc:194] Registered new tserver with Master: 1149b5eb4e3b4e79aa969ee23a4f09b8 (127.12.34.194:33689)
I20260926 15:13:56.190553 32315 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.34.194:49491
W20260926 15:13:56.499600 32635 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:13:56.499920 32635 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:13:56.500087 32635 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:13:56.507144 32635 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:13:56.507288 32635 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:13:56.507483 32635 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.34.195
I20260926 15:13:56.516482 32635 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.34.212:38985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790435318278681-12427-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790435318278681-12427-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.34.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790435318278681-12427-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.34.195
--webserver_port=0
--tserver_master_addrs=127.12.34.254:38865
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.34.195
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790435318278681-12427-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:13:56.517766 32635 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:13:56.519552 32635 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:13:56.532755 32641 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:13:56.532727 32640 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:13:56.532976 32643 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:13:56.533553 32635 server_base.cc:1034] running on GCE node
I20260926 15:13:56.534173 32635 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:13:56.535140 32635 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:13:56.536394 32635 hybrid_clock.cc:648] HybridClock initialized: now 1790435636536366 us; error 56 us; skew 500 ppm
I20260926 15:13:56.536761 32635 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 15:13:56.539153 32635 webserver.cc:460] Webserver started at http://127.12.34.195:43531/ using document root <none> and password file <none>
I20260926 15:13:56.539844 32635 fs_manager.cc:359] Metadata directory not provided
I20260926 15:13:56.539980 32635 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:13:56.540313 32635 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:13:56.543444 32635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790435318278681-12427-0/minicluster-data/ts-2/data/instance:
uuid: "b9eccc789b0d410581656ed3ca32d122"
format_stamp: "Formatted at 2026-09-26 15:13:56 on dist-test-slave-865v"
server_key: "268fa7fcd9cb6ad63dc0dd9565d0489a"
server_key_iv: "a1dc84b95043f767e206c4e83f891c0c"
server_key_version: "encryptionkey@0"
I20260926 15:13:56.544380 32635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790435318278681-12427-0/minicluster-data/ts-2/wal/instance:
uuid: "b9eccc789b0d410581656ed3ca32d122"
format_stamp: "Formatted at 2026-09-26 15:13:56 on dist-test-slave-865v"
server_key: "268fa7fcd9cb6ad63dc0dd9565d0489a"
server_key_iv: "a1dc84b95043f767e206c4e83f891c0c"
server_key_version: "encryptionkey@0"
I20260926 15:13:56.549588 32635 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 15:13:56.553169 32649 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:13:56.554436 32635 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 15:13:56.554613 32635 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790435318278681-12427-0/minicluster-data/ts-2/data,/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790435318278681-12427-0/minicluster-data/ts-2/wal
uuid: "b9eccc789b0d410581656ed3ca32d122"
format_stamp: "Formatted at 2026-09-26 15:13:56 on dist-test-slave-865v"
server_key: "268fa7fcd9cb6ad63dc0dd9565d0489a"
server_key_iv: "a1dc84b95043f767e206c4e83f891c0c"
server_key_version: "encryptionkey@0"
I20260926 15:13:56.554832 32635 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790435318278681-12427-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790435318278681-12427-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790435318278681-12427-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:13:56.567147 32635 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:13:56.567883 32635 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:13:56.568182 32635 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:13:56.569275 32635 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:13:56.571481 32635 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:13:56.571594 32635 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:13:56.571712 32635 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:13:56.571790 32635 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:13:56.608212 32635 rpc_server.cc:307] RPC server started. Bound to: 127.12.34.195:42341
I20260926 15:13:56.608265 32761 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.34.195:42341 every 8 connection(s)
I20260926 15:13:56.609766 32635 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790435318278681-12427-0/minicluster-data/ts-2/data/info.pb
I20260926 15:13:56.613473 12427 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8uoJGv/build/asan/bin/kudu as pid 32635
I20260926 15:13:56.613662 12427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790435318278681-12427-0/minicluster-data/ts-2/wal/instance
I20260926 15:13:56.614167 12427 external_mini_cluster.cc:1428] Setting key 0ca58dd6f3e140fc17eaf7bf4ffa62b0
I20260926 15:13:56.622526 32503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:56.612343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.195:56927 (local address 127.12.34.254:38865)
0926 15:13:56.612509 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0926 15:13:56.612514 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:13:56.613847 (+ 1333us) server_negotiation.cc:407] Connection header received
0926 15:13:56.616049 (+ 2202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:13:56.616054 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:13:56.616144 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:13:56.616248 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:13:56.617499 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:56.618266 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:56.619800 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:56.620211 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:56.620355 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:13:56.621699 (+ 1344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:13:56.621705 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:13:56.621710 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:13:56.621737 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:13:56.621767 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:13:56.621779 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:13:56.621875 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 15:13:56.622101 (+ 226us) server_negotiation.cc:299] Negotiation successful
0926 15:13:56.622193 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 15:13:56.623392 32764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:56.612750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:38865 (local address 127.12.34.195:56927)
0926 15:13:56.613680 (+ 930us) negotiation.cc:107] Waiting for socket to connect
0926 15:13:56.613756 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0926 15:13:56.615707 (+ 1951us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:13:56.616482 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:13:56.616490 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:13:56.616617 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:13:56.617343 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:13:56.617362 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:13:56.618423 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:13:56.618429 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:13:56.619657 (+ 1228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:13:56.619669 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:13:56.620429 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:13:56.620436 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:13:56.620658 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:13:56.621407 (+ 749us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 15:13:56.621409 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:13:56.621443 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 15:13:56.621512 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:13:56.621879 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:13:56.621892 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:13:56.621901 (+ 9us) client_negotiation.cc:769] Sending connection context
0926 15:13:56.622073 (+ 172us) client_negotiation.cc:240] Negotiation successful
0926 15:13:56.622419 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":313,"threads_started":1}
I20260926 15:13:56.624626 32762 heartbeater.cc:347] Connected to a master server at 127.12.34.254:38865
I20260926 15:13:56.624961 32762 heartbeater.cc:464] Registering TS with master...
I20260926 15:13:56.625631 32762 heartbeater.cc:511] Master 127.12.34.254:38865 requested a full tablet report, sending...
I20260926 15:13:56.626964 32315 ts_manager.cc:194] Registered new tserver with Master: b9eccc789b0d410581656ed3ca32d122 (127.12.34.195:42341)
I20260926 15:13:56.627898 32315 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.34.195:56927
I20260926 15:13:56.629359 12427 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 15:13:56.638983 32503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:56.633243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41870 (local address 127.12.34.254:38865)
0926 15:13:56.633419 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0926 15:13:56.633423 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:13:56.633840 (+ 417us) server_negotiation.cc:407] Connection header received
0926 15:13:56.634226 (+ 386us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:13:56.634231 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:13:56.634310 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:13:56.634394 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:13:56.634838 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:56.635505 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:56.636890 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:56.637374 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:56.637468 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:13:56.638430 (+ 962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:13:56.638434 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:13:56.638437 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:13:56.638453 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:13:56.638481 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:13:56.638493 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:13:56.638566 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 15:13:56.638707 (+ 141us) server_negotiation.cc:299] Negotiation successful
0926 15:13:56.638774 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 15:13:56.642904 32315 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41870:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 15:13:56.649698 32315 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 15:13:56.675904 309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:56.662139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59280 (local address 127.12.34.193:40709)
0926 15:13:56.663033 (+ 894us) server_negotiation.cc:206] Beginning negotiation
0926 15:13:56.663044 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 15:13:56.663572 (+ 528us) server_negotiation.cc:407] Connection header received
0926 15:13:56.663887 (+ 315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:13:56.663898 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:13:56.664177 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:13:56.664371 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:13:56.665517 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:56.667437 (+ 1920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:56.671954 (+ 4517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:56.673994 (+ 2040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:56.674132 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:13:56.674470 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0926 15:13:56.675370 (+ 900us) server_negotiation.cc:299] Negotiation successful
0926 15:13:56.675574 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":746,"thread_start_us":640,"threads_started":1}
I20260926 15:13:56.676645 310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:56.661578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.193:40709 (local address 127.0.0.1:59280)
0926 15:13:56.663461 (+ 1883us) negotiation.cc:107] Waiting for socket to connect
0926 15:13:56.663479 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 15:13:56.663669 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:13:56.664727 (+ 1058us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:13:56.664738 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:13:56.664776 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:13:56.665276 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:13:56.665293 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:13:56.667671 (+ 2378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:13:56.667680 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:13:56.671689 (+ 4009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:13:56.671710 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:13:56.674714 (+ 3004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:13:56.674723 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:13:56.675020 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:13:56.675090 (+ 70us) client_negotiation.cc:769] Sending connection context
0926 15:13:56.676101 (+ 1011us) client_negotiation.cc:240] Negotiation successful
0926 15:13:56.676301 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1734,"thread_start_us":383,"threads_started":1}
I20260926 15:13:56.680670 32436 tablet_service.cc:1467] Processing CreateTablet for tablet 6002d5f140fc44ae83f47d83deb7d30c (DEFAULT_TABLE table=test-table [id=9cf95f71babb475fbd2be1e01099d473]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:13:56.682179 32436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6002d5f140fc44ae83f47d83deb7d30c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:13:56.685047 307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:56.661452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.194:33689 (local address 127.0.0.1:43038)
0926 15:13:56.662327 (+ 875us) negotiation.cc:107] Waiting for socket to connect
0926 15:13:56.662376 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 15:13:56.662537 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:13:56.678225 (+ 15688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:13:56.678230 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:13:56.678263 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:13:56.678660 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:13:56.678672 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:13:56.679985 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:13:56.679992 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:13:56.682497 (+ 2505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:13:56.682512 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:13:56.684282 (+ 1770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:13:56.684286 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:13:56.684397 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:13:56.684420 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 15:13:56.684521 (+ 101us) client_negotiation.cc:240] Negotiation successful
0926 15:13:56.684650 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":643,"mutex_wait_us":110,"thread_start_us":15720,"threads_started":1}
I20260926 15:13:56.685137 308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:56.661785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43038 (local address 127.12.34.194:33689)
0926 15:13:56.677703 (+ 15918us) server_negotiation.cc:206] Beginning negotiation
0926 15:13:56.677713 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 15:13:56.677740 (+ 27us) server_negotiation.cc:407] Connection header received
0926 15:13:56.677832 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:13:56.677839 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:13:56.678035 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:13:56.678223 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:13:56.678854 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:56.679837 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:56.682737 (+ 2900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:56.684097 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:56.684244 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:13:56.684520 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0926 15:13:56.684587 (+ 67us) server_negotiation.cc:299] Negotiation successful
0926 15:13:56.684733 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15184,"thread_start_us":15053,"threads_started":1}
I20260926 15:13:56.687742 306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:56.662739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.195:42341 (local address 127.0.0.1:45072)
0926 15:13:56.676912 (+ 14173us) negotiation.cc:107] Waiting for socket to connect
0926 15:13:56.676932 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 15:13:56.677063 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:13:56.677962 (+ 899us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:13:56.677968 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:13:56.677999 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:13:56.678295 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:13:56.678307 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:13:56.680175 (+ 1868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:13:56.680187 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:13:56.682499 (+ 2312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:13:56.682512 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:13:56.686966 (+ 4454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:13:56.686973 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:13:56.687145 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:13:56.687193 (+ 48us) client_negotiation.cc:769] Sending connection context
0926 15:13:56.687297 (+ 104us) client_negotiation.cc:240] Negotiation successful
0926 15:13:56.687436 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14018,"mutex_wait_us":69,"thread_start_us":815,"threads_started":1}
I20260926 15:13:56.687883 311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:56.663275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45072 (local address 127.12.34.195:42341)
0926 15:13:56.675851 (+ 12576us) server_negotiation.cc:206] Beginning negotiation
0926 15:13:56.675863 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 15:13:56.677117 (+ 1254us) server_negotiation.cc:407] Connection header received
0926 15:13:56.677244 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:13:56.677255 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:13:56.677511 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:13:56.677709 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:13:56.678515 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:56.680043 (+ 1528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:56.685370 (+ 5327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:56.686771 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:56.686901 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:13:56.687143 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0926 15:13:56.687353 (+ 210us) server_negotiation.cc:299] Negotiation successful
0926 15:13:56.687515 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12376,"thread_start_us":12587,"threads_started":1}
I20260926 15:13:56.689524 32567 tablet_service.cc:1467] Processing CreateTablet for tablet 6002d5f140fc44ae83f47d83deb7d30c (DEFAULT_TABLE table=test-table [id=9cf95f71babb475fbd2be1e01099d473]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:13:56.690737 32567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6002d5f140fc44ae83f47d83deb7d30c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:13:56.691370 32697 tablet_service.cc:1467] Processing CreateTablet for tablet 6002d5f140fc44ae83f47d83deb7d30c (DEFAULT_TABLE table=test-table [id=9cf95f71babb475fbd2be1e01099d473]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:13:56.692726 32697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6002d5f140fc44ae83f47d83deb7d30c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:13:56.697948 312 tablet_bootstrap.cc:492] T 6002d5f140fc44ae83f47d83deb7d30c P d5f58c067f4348428a28b86bed674028: Bootstrap starting.
I20260926 15:13:56.701516 312 tablet_bootstrap.cc:654] T 6002d5f140fc44ae83f47d83deb7d30c P d5f58c067f4348428a28b86bed674028: Neither blocks nor log segments found. Creating new log.
I20260926 15:13:56.702796 312 log.cc:824] T 6002d5f140fc44ae83f47d83deb7d30c P d5f58c067f4348428a28b86bed674028: Log is configured to *not* fsync() on all Append() calls
I20260926 15:13:56.702812 313 tablet_bootstrap.cc:492] T 6002d5f140fc44ae83f47d83deb7d30c P 1149b5eb4e3b4e79aa969ee23a4f09b8: Bootstrap starting.
I20260926 15:13:56.705926 315 tablet_bootstrap.cc:492] T 6002d5f140fc44ae83f47d83deb7d30c P b9eccc789b0d410581656ed3ca32d122: Bootstrap starting.
I20260926 15:13:56.706282 313 tablet_bootstrap.cc:654] T 6002d5f140fc44ae83f47d83deb7d30c P 1149b5eb4e3b4e79aa969ee23a4f09b8: Neither blocks nor log segments found. Creating new log.
I20260926 15:13:56.706092 312 tablet_bootstrap.cc:492] T 6002d5f140fc44ae83f47d83deb7d30c P d5f58c067f4348428a28b86bed674028: No bootstrap required, opened a new log
I20260926 15:13:56.706724 312 ts_tablet_manager.cc:1397] T 6002d5f140fc44ae83f47d83deb7d30c P d5f58c067f4348428a28b86bed674028: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260926 15:13:56.707713 313 log.cc:824] T 6002d5f140fc44ae83f47d83deb7d30c P 1149b5eb4e3b4e79aa969ee23a4f09b8: Log is configured to *not* fsync() on all Append() calls
I20260926 15:13:56.709323 315 tablet_bootstrap.cc:654] T 6002d5f140fc44ae83f47d83deb7d30c P b9eccc789b0d410581656ed3ca32d122: Neither blocks nor log segments found. Creating new log.
I20260926 15:13:56.710124 313 tablet_bootstrap.cc:492] T 6002d5f140fc44ae83f47d83deb7d30c P 1149b5eb4e3b4e79aa969ee23a4f09b8: No bootstrap required, opened a new log
I20260926 15:13:56.710366 313 ts_tablet_manager.cc:1397] T 6002d5f140fc44ae83f47d83deb7d30c P 1149b5eb4e3b4e79aa969ee23a4f09b8: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.005s
I20260926 15:13:56.710436 315 log.cc:824] T 6002d5f140fc44ae83f47d83deb7d30c P b9eccc789b0d410581656ed3ca32d122: Log is configured to *not* fsync() on all Append() calls
I20260926 15:13:56.713234 315 tablet_bootstrap.cc:492] T 6002d5f140fc44ae83f47d83deb7d30c P b9eccc789b0d410581656ed3ca32d122: No bootstrap required, opened a new log
I20260926 15:13:56.713470 315 ts_tablet_manager.cc:1397] T 6002d5f140fc44ae83f47d83deb7d30c P b9eccc789b0d410581656ed3ca32d122: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.005s
I20260926 15:13:56.716921 312 raft_consensus.cc:348] T 6002d5f140fc44ae83f47d83deb7d30c P d5f58c067f4348428a28b86bed674028 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1149b5eb4e3b4e79aa969ee23a4f09b8" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 33689 } } peers { permanent_uuid: "d5f58c067f4348428a28b86bed674028" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 40709 } } peers { permanent_uuid: "b9eccc789b0d410581656ed3ca32d122" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 42341 } }
I20260926 15:13:56.717412 312 raft_consensus.cc:374] T 6002d5f140fc44ae83f47d83deb7d30c P d5f58c067f4348428a28b86bed674028 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:13:56.717514 312 raft_consensus.cc:729] T 6002d5f140fc44ae83f47d83deb7d30c P d5f58c067f4348428a28b86bed674028 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5f58c067f4348428a28b86bed674028, State: Initialized, Role: FOLLOWER
I20260926 15:13:56.718097 312 consensus_queue.cc:260] T 6002d5f140fc44ae83f47d83deb7d30c P d5f58c067f4348428a28b86bed674028 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1149b5eb4e3b4e79aa969ee23a4f09b8" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 33689 } } peers { permanent_uuid: "d5f58c067f4348428a28b86bed674028" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 40709 } } peers { permanent_uuid: "b9eccc789b0d410581656ed3ca32d122" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 42341 } }
I20260926 15:13:56.719285 32501 heartbeater.cc:503] Master 127.12.34.254:38865 was elected leader, sending a full tablet report...
I20260926 15:13:56.719470 313 raft_consensus.cc:348] T 6002d5f140fc44ae83f47d83deb7d30c P 1149b5eb4e3b4e79aa969ee23a4f09b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1149b5eb4e3b4e79aa969ee23a4f09b8" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 33689 } } peers { permanent_uuid: "d5f58c067f4348428a28b86bed674028" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 40709 } } peers { permanent_uuid: "b9eccc789b0d410581656ed3ca32d122" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 42341 } }
I20260926 15:13:56.719826 313 raft_consensus.cc:374] T 6002d5f140fc44ae83f47d83deb7d30c P 1149b5eb4e3b4e79aa969ee23a4f09b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:13:56.719834 312 ts_tablet_manager.cc:1428] T 6002d5f140fc44ae83f47d83deb7d30c P d5f58c067f4348428a28b86bed674028: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260926 15:13:56.719944 313 raft_consensus.cc:729] T 6002d5f140fc44ae83f47d83deb7d30c P 1149b5eb4e3b4e79aa969ee23a4f09b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1149b5eb4e3b4e79aa969ee23a4f09b8, State: Initialized, Role: FOLLOWER
I20260926 15:13:56.720527 313 consensus_queue.cc:260] T 6002d5f140fc44ae83f47d83deb7d30c P 1149b5eb4e3b4e79aa969ee23a4f09b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1149b5eb4e3b4e79aa969ee23a4f09b8" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 33689 } } peers { permanent_uuid: "d5f58c067f4348428a28b86bed674028" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 40709 } } peers { permanent_uuid: "b9eccc789b0d410581656ed3ca32d122" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 42341 } }
I20260926 15:13:56.721697 32632 heartbeater.cc:503] Master 127.12.34.254:38865 was elected leader, sending a full tablet report...
I20260926 15:13:56.722355 313 ts_tablet_manager.cc:1428] T 6002d5f140fc44ae83f47d83deb7d30c P 1149b5eb4e3b4e79aa969ee23a4f09b8: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260926 15:13:56.724526 315 raft_consensus.cc:348] T 6002d5f140fc44ae83f47d83deb7d30c P b9eccc789b0d410581656ed3ca32d122 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1149b5eb4e3b4e79aa969ee23a4f09b8" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 33689 } } peers { permanent_uuid: "d5f58c067f4348428a28b86bed674028" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 40709 } } peers { permanent_uuid: "b9eccc789b0d410581656ed3ca32d122" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 42341 } }
I20260926 15:13:56.725057 315 raft_consensus.cc:374] T 6002d5f140fc44ae83f47d83deb7d30c P b9eccc789b0d410581656ed3ca32d122 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:13:56.725236 315 raft_consensus.cc:729] T 6002d5f140fc44ae83f47d83deb7d30c P b9eccc789b0d410581656ed3ca32d122 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9eccc789b0d410581656ed3ca32d122, State: Initialized, Role: FOLLOWER
I20260926 15:13:56.725883 315 consensus_queue.cc:260] T 6002d5f140fc44ae83f47d83deb7d30c P b9eccc789b0d410581656ed3ca32d122 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1149b5eb4e3b4e79aa969ee23a4f09b8" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 33689 } } peers { permanent_uuid: "d5f58c067f4348428a28b86bed674028" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 40709 } } peers { permanent_uuid: "b9eccc789b0d410581656ed3ca32d122" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 42341 } }
W20260926 15:13:56.726719 32502 tablet.cc:2369] T 6002d5f140fc44ae83f47d83deb7d30c P d5f58c067f4348428a28b86bed674028: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 15:13:56.733301 32762 heartbeater.cc:503] Master 127.12.34.254:38865 was elected leader, sending a full tablet report...
I20260926 15:13:56.734483 315 ts_tablet_manager.cc:1428] T 6002d5f140fc44ae83f47d83deb7d30c P b9eccc789b0d410581656ed3ca32d122: Time spent starting tablet: real 0.021s user 0.011s sys 0.008s
I20260926 15:13:56.768568 319 raft_consensus.cc:482] T 6002d5f140fc44ae83f47d83deb7d30c P 1149b5eb4e3b4e79aa969ee23a4f09b8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 15:13:56.768977 319 raft_consensus.cc:504] T 6002d5f140fc44ae83f47d83deb7d30c P 1149b5eb4e3b4e79aa969ee23a4f09b8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1149b5eb4e3b4e79aa969ee23a4f09b8" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 33689 } } peers { permanent_uuid: "d5f58c067f4348428a28b86bed674028" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 40709 } } peers { permanent_uuid: "b9eccc789b0d410581656ed3ca32d122" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 42341 } }
I20260926 15:13:56.770828 319 leader_election.cc:288] T 6002d5f140fc44ae83f47d83deb7d30c P 1149b5eb4e3b4e79aa969ee23a4f09b8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d5f58c067f4348428a28b86bed674028 (127.12.34.193:40709), b9eccc789b0d410581656ed3ca32d122 (127.12.34.195:42341)
I20260926 15:13:56.778273 309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:56.771396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.194:50369 (local address 127.12.34.193:40709)
0926 15:13:56.771656 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0926 15:13:56.771660 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:13:56.772239 (+ 579us) server_negotiation.cc:407] Connection header received
0926 15:13:56.772454 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:13:56.772460 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:13:56.772553 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:13:56.772659 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:13:56.773406 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:56.774080 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:56.776442 (+ 2362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:56.777308 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:56.777401 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:13:56.777450 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0926 15:13:56.777881 (+ 431us) server_negotiation.cc:299] Negotiation successful
0926 15:13:56.778001 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128}
I20260926 15:13:56.778321 321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:56.771332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.193:40709 (local address 127.12.34.194:50369)
0926 15:13:56.772092 (+ 760us) negotiation.cc:107] Waiting for socket to connect
0926 15:13:56.772123 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0926 15:13:56.772269 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:13:56.772800 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:13:56.772806 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:13:56.772839 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:13:56.773243 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:13:56.773253 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:13:56.774207 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:13:56.774215 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:13:56.776047 (+ 1832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:13:56.776060 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:13:56.777583 (+ 1523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:13:56.777587 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:13:56.777720 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:13:56.777740 (+ 20us) client_negotiation.cc:769] Sending connection context
0926 15:13:56.777849 (+ 109us) client_negotiation.cc:240] Negotiation successful
0926 15:13:56.777983 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":611,"thread_start_us":416,"threads_started":1}
I20260926 15:13:56.778570 311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:56.772498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.34.194:58633 (local address 127.12.34.195:42341)
0926 15:13:56.772704 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0926 15:13:56.772710 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:13:56.773060 (+ 350us) server_negotiation.cc:407] Connection header received
0926 15:13:56.773300 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:13:56.773305 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:13:56.773373 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:13:56.773468 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:13:56.773948 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:56.774623 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:56.776875 (+ 2252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:56.777680 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:56.777863 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:13:56.777926 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 15:13:56.778211 (+ 285us) server_negotiation.cc:299] Negotiation successful
0926 15:13:56.778332 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260926 15:13:56.778659 322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:56.772207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.195:42341 (local address 127.12.34.194:58633)
0926 15:13:56.772943 (+ 736us) negotiation.cc:107] Waiting for socket to connect
0926 15:13:56.772960 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 15:13:56.773135 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:13:56.773479 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:13:56.773484 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:13:56.773506 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:13:56.773800 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:13:56.773808 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:13:56.774765 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:13:56.774772 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:13:56.776684 (+ 1912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:13:56.776695 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:13:56.777871 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:13:56.777877 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:13:56.778047 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:13:56.778069 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 15:13:56.778172 (+ 103us) client_negotiation.cc:240] Negotiation successful
0926 15:13:56.778306 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":623,"mutex_wait_us":6,"thread_start_us":453,"threads_started":1}
I20260926 15:13:56.779500 32456 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6002d5f140fc44ae83f47d83deb7d30c" candidate_uuid: "1149b5eb4e3b4e79aa969ee23a4f09b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5f58c067f4348428a28b86bed674028" is_pre_election: true
I20260926 15:13:56.780021 32456 raft_consensus.cc:2454] T 6002d5f140fc44ae83f47d83deb7d30c P d5f58c067f4348428a28b86bed674028 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1149b5eb4e3b4e79aa969ee23a4f09b8 in term 0.
I20260926 15:13:56.779937 32717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6002d5f140fc44ae83f47d83deb7d30c" candidate_uuid: "1149b5eb4e3b4e79aa969ee23a4f09b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9eccc789b0d410581656ed3ca32d122" is_pre_election: true
I20260926 15:13:56.780390 32717 raft_consensus.cc:2454] T 6002d5f140fc44ae83f47d83deb7d30c P b9eccc789b0d410581656ed3ca32d122 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1149b5eb4e3b4e79aa969ee23a4f09b8 in term 0.
I20260926 15:13:56.780718 32521 leader_election.cc:302] T 6002d5f140fc44ae83f47d83deb7d30c P 1149b5eb4e3b4e79aa969ee23a4f09b8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1149b5eb4e3b4e79aa969ee23a4f09b8, d5f58c067f4348428a28b86bed674028; no voters:
I20260926 15:13:56.781131 319 raft_consensus.cc:2781] T 6002d5f140fc44ae83f47d83deb7d30c P 1149b5eb4e3b4e79aa969ee23a4f09b8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 15:13:56.781281 319 raft_consensus.cc:482] T 6002d5f140fc44ae83f47d83deb7d30c P 1149b5eb4e3b4e79aa969ee23a4f09b8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 15:13:56.781386 319 raft_consensus.cc:3037] T 6002d5f140fc44ae83f47d83deb7d30c P 1149b5eb4e3b4e79aa969ee23a4f09b8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:13:56.783522 319 raft_consensus.cc:504] T 6002d5f140fc44ae83f47d83deb7d30c P 1149b5eb4e3b4e79aa969ee23a4f09b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1149b5eb4e3b4e79aa969ee23a4f09b8" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 33689 } } peers { permanent_uuid: "d5f58c067f4348428a28b86bed674028" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 40709 } } peers { permanent_uuid: "b9eccc789b0d410581656ed3ca32d122" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 42341 } }
I20260926 15:13:56.784133 319 leader_election.cc:288] T 6002d5f140fc44ae83f47d83deb7d30c P 1149b5eb4e3b4e79aa969ee23a4f09b8 [CANDIDATE]: Term 1 election: Requested vote from peers d5f58c067f4348428a28b86bed674028 (127.12.34.193:40709), b9eccc789b0d410581656ed3ca32d122 (127.12.34.195:42341)
I20260926 15:13:56.784529 32456 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6002d5f140fc44ae83f47d83deb7d30c" candidate_uuid: "1149b5eb4e3b4e79aa969ee23a4f09b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5f58c067f4348428a28b86bed674028"
I20260926 15:13:56.784603 32717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6002d5f140fc44ae83f47d83deb7d30c" candidate_uuid: "1149b5eb4e3b4e79aa969ee23a4f09b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9eccc789b0d410581656ed3ca32d122"
I20260926 15:13:56.784724 32456 raft_consensus.cc:3037] T 6002d5f140fc44ae83f47d83deb7d30c P d5f58c067f4348428a28b86bed674028 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:13:56.784861 32717 raft_consensus.cc:3037] T 6002d5f140fc44ae83f47d83deb7d30c P b9eccc789b0d410581656ed3ca32d122 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:13:56.787079 32456 raft_consensus.cc:2454] T 6002d5f140fc44ae83f47d83deb7d30c P d5f58c067f4348428a28b86bed674028 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1149b5eb4e3b4e79aa969ee23a4f09b8 in term 1.
I20260926 15:13:56.787448 32521 leader_election.cc:302] T 6002d5f140fc44ae83f47d83deb7d30c P 1149b5eb4e3b4e79aa969ee23a4f09b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1149b5eb4e3b4e79aa969ee23a4f09b8, d5f58c067f4348428a28b86bed674028; no voters:
I20260926 15:13:56.787642 32717 raft_consensus.cc:2454] T 6002d5f140fc44ae83f47d83deb7d30c P b9eccc789b0d410581656ed3ca32d122 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1149b5eb4e3b4e79aa969ee23a4f09b8 in term 1.
I20260926 15:13:56.787725 319 raft_consensus.cc:2781] T 6002d5f140fc44ae83f47d83deb7d30c P 1149b5eb4e3b4e79aa969ee23a4f09b8 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:13:56.788236 319 raft_consensus.cc:686] T 6002d5f140fc44ae83f47d83deb7d30c P 1149b5eb4e3b4e79aa969ee23a4f09b8 [term 1 LEADER]: Becoming Leader. State: Replica: 1149b5eb4e3b4e79aa969ee23a4f09b8, State: Running, Role: LEADER
I20260926 15:13:56.788787 319 consensus_queue.cc:237] T 6002d5f140fc44ae83f47d83deb7d30c P 1149b5eb4e3b4e79aa969ee23a4f09b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1149b5eb4e3b4e79aa969ee23a4f09b8" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 33689 } } peers { permanent_uuid: "d5f58c067f4348428a28b86bed674028" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 40709 } } peers { permanent_uuid: "b9eccc789b0d410581656ed3ca32d122" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 42341 } }
I20260926 15:13:56.793757 32315 catalog_manager.cc:5495] T 6002d5f140fc44ae83f47d83deb7d30c P 1149b5eb4e3b4e79aa969ee23a4f09b8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1149b5eb4e3b4e79aa969ee23a4f09b8 (127.12.34.194). New cstate: current_term: 1 leader_uuid: "1149b5eb4e3b4e79aa969ee23a4f09b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1149b5eb4e3b4e79aa969ee23a4f09b8" member_type: VOTER last_known_addr { host: "127.12.34.194" port: 33689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d5f58c067f4348428a28b86bed674028" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 40709 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9eccc789b0d410581656ed3ca32d122" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 42341 } health_report { overall_health: UNKNOWN } } }
I20260926 15:13:56.811318 308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:56.806382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43048 (local address 127.12.34.194:33689)
0926 15:13:56.806556 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0926 15:13:56.806564 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 15:13:56.806582 (+ 18us) server_negotiation.cc:407] Connection header received
0926 15:13:56.806689 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:13:56.806692 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:13:56.806761 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:13:56.806907 (+ 146us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 15:13:56.807259 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:56.807943 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:56.809338 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:56.809747 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:56.809885 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:13:56.810325 (+ 440us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 15:13:56.810773 (+ 448us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 15:13:56.810931 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0926 15:13:56.810988 (+ 57us) server_negotiation.cc:299] Negotiation successful
0926 15:13:56.811061 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 15:13:56.818836 32717 raft_consensus.cc:1261] T 6002d5f140fc44ae83f47d83deb7d30c P b9eccc789b0d410581656ed3ca32d122 [term 1 FOLLOWER]: Refusing update from remote peer 1149b5eb4e3b4e79aa969ee23a4f09b8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:13:56.818845 32456 raft_consensus.cc:1261] T 6002d5f140fc44ae83f47d83deb7d30c P d5f58c067f4348428a28b86bed674028 [term 1 FOLLOWER]: Refusing update from remote peer 1149b5eb4e3b4e79aa969ee23a4f09b8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:13:56.819778 323 consensus_queue.cc:1035] T 6002d5f140fc44ae83f47d83deb7d30c P 1149b5eb4e3b4e79aa969ee23a4f09b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9eccc789b0d410581656ed3ca32d122" member_type: VOTER last_known_addr { host: "127.12.34.195" port: 42341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:13:56.820112 319 consensus_queue.cc:1035] T 6002d5f140fc44ae83f47d83deb7d30c P 1149b5eb4e3b4e79aa969ee23a4f09b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5f58c067f4348428a28b86bed674028" member_type: VOTER last_known_addr { host: "127.12.34.193" port: 40709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:13:56.832683 326 mvcc.cc:205] Tried to move back new op lower bound from 7333624368400695296 to 7333624368291246080. Current Snapshot: MvccSnapshot[applied={T|T < 7333624368400695296}]
I20260926 15:13:56.847616 32315 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41870:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 15:13:56.847882 32315 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41870:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 15:13:56.851433 32315 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8522f440d0e240dbac8cf905b2190263: Sending DeleteTablet for 3 replicas of tablet 6002d5f140fc44ae83f47d83deb7d30c
I20260926 15:13:56.852700 32567 tablet_service.cc:1514] Processing DeleteTablet for tablet 6002d5f140fc44ae83f47d83deb7d30c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:13:56 UTC) from {username='slave'} at 127.0.0.1:43038
I20260926 15:13:56.852748 32697 tablet_service.cc:1514] Processing DeleteTablet for tablet 6002d5f140fc44ae83f47d83deb7d30c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:13:56 UTC) from {username='slave'} at 127.0.0.1:45072
I20260926 15:13:56.853829 32436 tablet_service.cc:1514] Processing DeleteTablet for tablet 6002d5f140fc44ae83f47d83deb7d30c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:13:56 UTC) from {username='slave'} at 127.0.0.1:59280
I20260926 15:13:56.854676 12427 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 32374
I20260926 15:13:56.864006 338 tablet_replica.cc:330] T 6002d5f140fc44ae83f47d83deb7d30c P d5f58c067f4348428a28b86bed674028: stopping tablet replica
I20260926 15:13:56.865437 338 raft_consensus.cc:2229] T 6002d5f140fc44ae83f47d83deb7d30c P d5f58c067f4348428a28b86bed674028 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 15:13:56.866096 337 tablet_replica.cc:330] T 6002d5f140fc44ae83f47d83deb7d30c P 1149b5eb4e3b4e79aa969ee23a4f09b8: stopping tablet replica
I20260926 15:13:56.866652 337 raft_consensus.cc:2229] T 6002d5f140fc44ae83f47d83deb7d30c P 1149b5eb4e3b4e79aa969ee23a4f09b8 [term 1 LEADER]: Raft consensus shutting down.
I20260926 15:13:56.866809 338 raft_consensus.cc:2258] T 6002d5f140fc44ae83f47d83deb7d30c P d5f58c067f4348428a28b86bed674028 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:13:56.867449 337 raft_consensus.cc:2258] T 6002d5f140fc44ae83f47d83deb7d30c P 1149b5eb4e3b4e79aa969ee23a4f09b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:13:56.867643 336 tablet_replica.cc:330] T 6002d5f140fc44ae83f47d83deb7d30c P b9eccc789b0d410581656ed3ca32d122: stopping tablet replica
I20260926 15:13:56.868275 336 raft_consensus.cc:2229] T 6002d5f140fc44ae83f47d83deb7d30c P b9eccc789b0d410581656ed3ca32d122 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 15:13:56.868878 336 raft_consensus.cc:2258] T 6002d5f140fc44ae83f47d83deb7d30c P b9eccc789b0d410581656ed3ca32d122 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:13:56.869612 337 ts_tablet_manager.cc:1905] T 6002d5f140fc44ae83f47d83deb7d30c P 1149b5eb4e3b4e79aa969ee23a4f09b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:13:56.870244 338 ts_tablet_manager.cc:1905] T 6002d5f140fc44ae83f47d83deb7d30c P d5f58c067f4348428a28b86bed674028: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:13:56.872170 336 ts_tablet_manager.cc:1905] T 6002d5f140fc44ae83f47d83deb7d30c P b9eccc789b0d410581656ed3ca32d122: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:13:56.871981 309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:56.855982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59286 (local address 127.12.34.193:40709)
0926 15:13:56.856154 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0926 15:13:56.856160 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:13:56.861583 (+ 5423us) server_negotiation.cc:407] Connection header received
0926 15:13:56.861814 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:13:56.861819 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:13:56.861935 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:13:56.862035 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:13:56.864109 (+ 2074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:56.865303 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:56.867184 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:56.867871 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:56.867969 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:13:56.870236 (+ 2267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:13:56.870247 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:13:56.870265 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:13:56.871035 (+ 770us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:13:56.871096 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:13:56.871120 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:13:56.871212 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 15:13:56.871510 (+ 298us) server_negotiation.cc:299] Negotiation successful
0926 15:13:56.871621 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 15:13:56.872961 32496 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:13:56.877146 337 ts_tablet_manager.cc:1918] T 6002d5f140fc44ae83f47d83deb7d30c P 1149b5eb4e3b4e79aa969ee23a4f09b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 15:13:56.877358 337 log.cc:1196] T 6002d5f140fc44ae83f47d83deb7d30c P 1149b5eb4e3b4e79aa969ee23a4f09b8: Deleting WAL directory at /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790435318278681-12427-0/minicluster-data/ts-1/wal/wals/6002d5f140fc44ae83f47d83deb7d30c
I20260926 15:13:56.878006 337 ts_tablet_manager.cc:1939] T 6002d5f140fc44ae83f47d83deb7d30c P 1149b5eb4e3b4e79aa969ee23a4f09b8: Deleting consensus metadata
I20260926 15:13:56.879909 32302 catalog_manager.cc:4841] TS 1149b5eb4e3b4e79aa969ee23a4f09b8 (127.12.34.194:33689): tablet 6002d5f140fc44ae83f47d83deb7d30c (table test-table [id=9cf95f71babb475fbd2be1e01099d473]) successfully deleted
I20260926 15:13:56.882366 336 ts_tablet_manager.cc:1918] T 6002d5f140fc44ae83f47d83deb7d30c P b9eccc789b0d410581656ed3ca32d122: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 15:13:56.882717 336 log.cc:1196] T 6002d5f140fc44ae83f47d83deb7d30c P b9eccc789b0d410581656ed3ca32d122: Deleting WAL directory at /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790435318278681-12427-0/minicluster-data/ts-2/wal/wals/6002d5f140fc44ae83f47d83deb7d30c
I20260926 15:13:56.883589 336 ts_tablet_manager.cc:1939] T 6002d5f140fc44ae83f47d83deb7d30c P b9eccc789b0d410581656ed3ca32d122: Deleting consensus metadata
I20260926 15:13:56.885536 32300 catalog_manager.cc:4841] TS b9eccc789b0d410581656ed3ca32d122 (127.12.34.195:42341): tablet 6002d5f140fc44ae83f47d83deb7d30c (table test-table [id=9cf95f71babb475fbd2be1e01099d473]) successfully deleted
I20260926 15:13:57.004719 12427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8uoJGv/build/asan/bin/kudu with pid 32374
I20260926 15:13:57.023298 12427 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 32505
W20260926 15:13:57.023380 32300 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.34.193:40709 (error 108)
I20260926 15:13:57.025606 306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:57.025038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.193:40709 (local address 127.0.0.1:59300)
0926 15:13:57.025319 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0926 15:13:57.025433 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.193:40709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155}
W20260926 15:13:57.025866 32300 catalog_manager.cc:4584] TS d5f58c067f4348428a28b86bed674028 (127.12.34.193:40709): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6002d5f140fc44ae83f47d83deb7d30c: Network error: Client connection negotiation failed: client connection to 127.12.34.193:40709: connect: Connection refused (error 111)
I20260926 15:13:57.031351 308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:57.024549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43064 (local address 127.12.34.194:33689)
0926 15:13:57.024746 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0926 15:13:57.024751 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:13:57.024793 (+ 42us) server_negotiation.cc:407] Connection header received
0926 15:13:57.024877 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:13:57.024885 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:13:57.025002 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:13:57.025144 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:13:57.025572 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:57.026529 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:57.028374 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:57.028798 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:57.028910 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:13:57.030019 (+ 1109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:13:57.030027 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:13:57.030040 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:13:57.030616 (+ 576us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:13:57.030663 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:13:57.030683 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:13:57.030761 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 15:13:57.030919 (+ 158us) server_negotiation.cc:299] Negotiation successful
0926 15:13:57.030990 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 15:13:57.032034 32627 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:13:57.079881 306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:57.079266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.193:40709 (local address 127.0.0.1:59306)
0926 15:13:57.079521 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0926 15:13:57.079641 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.193:40709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260926 15:13:57.081018 306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:57.080629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.193:40709 (local address 127.0.0.1:59312)
0926 15:13:57.080794 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0926 15:13:57.080877 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.193:40709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260926 15:13:57.141568 306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:57.141058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.193:40709 (local address 127.0.0.1:59318)
0926 15:13:57.141305 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0926 15:13:57.141394 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.193:40709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
I20260926 15:13:57.142793 306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:57.142293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.193:40709 (local address 127.0.0.1:59326)
0926 15:13:57.142457 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0926 15:13:57.142608 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.193:40709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260926 15:13:57.147378 12427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8uoJGv/build/asan/bin/kudu with pid 32505
I20260926 15:13:57.167963 12427 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 32635
I20260926 15:13:57.175525 311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:57.168869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45088 (local address 127.12.34.195:42341)
0926 15:13:57.169177 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0926 15:13:57.169183 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:13:57.169210 (+ 27us) server_negotiation.cc:407] Connection header received
0926 15:13:57.169370 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:13:57.169374 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:13:57.169477 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:13:57.169595 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:13:57.170039 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:57.170844 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:57.172433 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:57.172892 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:57.173042 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:13:57.174197 (+ 1155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:13:57.174206 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:13:57.174220 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:13:57.174775 (+ 555us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:13:57.174822 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:13:57.174842 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:13:57.174965 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0926 15:13:57.175111 (+ 146us) server_negotiation.cc:299] Negotiation successful
0926 15:13:57.175228 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260926 15:13:57.176158 32757 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:13:57.223240 306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:57.222707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.193:40709 (local address 127.0.0.1:59340)
0926 15:13:57.222924 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0926 15:13:57.223044 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.193:40709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260926 15:13:57.224666 306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:57.224244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.193:40709 (local address 127.0.0.1:59346)
0926 15:13:57.224458 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0926 15:13:57.224544 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.193:40709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124}
I20260926 15:13:57.276382 12427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8uoJGv/build/asan/bin/kudu with pid 32635
I20260926 15:13:57.293736 12427 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 32284
I20260926 15:13:57.294940 32345 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:13:57.406245 12427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8uoJGv/build/asan/bin/kudu with pid 32284
2026-09-26T15:13:57Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (2732 ms)
[----------] 6 tests from EncryptionPolicyTest (21454 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790435318278681-12427-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 15:13:57 dist-test-slave-865v krb5kdc[347](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 15:13:57 dist-test-slave-865v krb5kdc[347](info): set up 2 sockets
Sep 26 15:13:57 dist-test-slave-865v krb5kdc[347](info): commencing operation
krb5kdc: starting...
W20260926 15:13:58.517701 12427 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790435318278681-12427-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790435318278681-12427-0/krb5kdc/test-user.keytab.
Sep 26 15:13:58 dist-test-slave-865v krb5kdc[347](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435638, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T15:13:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T15:13:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790435318278681-12427-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790435318278681-12427-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 15:13:58.773314 12427 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8uoJGv/build/asan/bin/kudu
/tmp/dist-test-task8uoJGv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790435318278681-12427-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790435318278681-12427-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790435318278681-12427-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37523
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790435318278681-12427-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37523
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790435318278681-12427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790435318278681-12427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790435318278681-12427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:13:59.094900 360 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:13:59.095170 360 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:13:59.095342 360 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:13:59.102465 360 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:13:59.102557 360 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:13:59.102665 360 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:13:59.102774 360 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:13:59.102859 360 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:13:59.112336 360 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790435318278681-12427-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790435318278681-12427-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37523
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790435318278681-12427-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37523
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790435318278681-12427-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:13:59.113588 360 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:13:59.114856 360 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:13:59.126411 365 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:13:59.126560 366 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:13:59.126438 368 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:13:59.127372 360 server_base.cc:1034] running on GCE node
I20260926 15:13:59.128041 360 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:13:59.129717 360 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:13:59.130976 360 hybrid_clock.cc:648] HybridClock initialized: now 1790435639130911 us; error 49 us; skew 500 ppm
I20260926 15:13:59.131374 360 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:13:59 dist-test-slave-865v krb5kdc[347](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435639, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:13:59.139662 360 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 15:13:59.141240 360 webserver.cc:460] Webserver started at http://127.0.0.1:44121/ using document root <none> and password file <none>
I20260926 15:13:59.141865 360 fs_manager.cc:359] Metadata directory not provided
I20260926 15:13:59.141965 360 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:13:59.142201 360 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:13:59.145059 360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790435318278681-12427-0/minicluster-data/master-0/data/instance:
uuid: "c86a16ef669d4827a4e269f15d1af196"
format_stamp: "Formatted at 2026-09-26 15:13:59 on dist-test-slave-865v"
server_key: "4525928acf9831f0df6282efdc509f50"
server_key_iv: "dfb54f1922322449a7833069e2535830"
server_key_version: "encryptionkey@0"
I20260926 15:13:59.145833 360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790435318278681-12427-0/minicluster-data/master-0/wal/instance:
uuid: "c86a16ef669d4827a4e269f15d1af196"
format_stamp: "Formatted at 2026-09-26 15:13:59 on dist-test-slave-865v"
server_key: "4525928acf9831f0df6282efdc509f50"
server_key_iv: "dfb54f1922322449a7833069e2535830"
server_key_version: "encryptionkey@0"
I20260926 15:13:59.150799 360 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 15:13:59.154142 375 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:13:59.155392 360 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 15:13:59.155577 360 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790435318278681-12427-0/minicluster-data/master-0/data,/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790435318278681-12427-0/minicluster-data/master-0/wal
uuid: "c86a16ef669d4827a4e269f15d1af196"
format_stamp: "Formatted at 2026-09-26 15:13:59 on dist-test-slave-865v"
server_key: "4525928acf9831f0df6282efdc509f50"
server_key_iv: "dfb54f1922322449a7833069e2535830"
server_key_version: "encryptionkey@0"
I20260926 15:13:59.155776 360 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790435318278681-12427-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790435318278681-12427-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790435318278681-12427-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:13:59.186396 360 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:13:59.190258 360 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:13:59.190481 360 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:13:59.208859 360 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37523
I20260926 15:13:59.208930 426 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37523 every 8 connection(s)
I20260926 15:13:59.210150 360 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
I20260926 15:13:59.213691 427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:13:59.216084 12427 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8uoJGv/build/asan/bin/kudu as pid 360
I20260926 15:13:59.216300 12427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790435318278681-12427-0/minicluster-data/master-0/wal/instance
I20260926 15:13:59.216914 12427 external_mini_cluster.cc:1428] Setting key 6f0fb8a0e5b21bdaf548a8c5f67ab57a
I20260926 15:13:59.225039 427 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c86a16ef669d4827a4e269f15d1af196: Bootstrap starting.
I20260926 15:13:59.228646 427 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c86a16ef669d4827a4e269f15d1af196: Neither blocks nor log segments found. Creating new log.
I20260926 15:13:59.229950 427 log.cc:824] T 00000000000000000000000000000000 P c86a16ef669d4827a4e269f15d1af196: Log is configured to *not* fsync() on all Append() calls
Sep 26 15:13:59 dist-test-slave-865v krb5kdc[347](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435638, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 15:13:59.232746 427 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c86a16ef669d4827a4e269f15d1af196: No bootstrap required, opened a new log
I20260926 15:13:59.239596 430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:59.218727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44863 (local address 127.0.0.1:37523)
0926 15:13:59.219466 (+ 739us) server_negotiation.cc:206] Beginning negotiation
0926 15:13:59.219477 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 15:13:59.219516 (+ 39us) server_negotiation.cc:407] Connection header received
0926 15:13:59.221160 (+ 1644us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:13:59.221179 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:13:59.221547 (+ 368us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:13:59.221834 (+ 287us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:13:59.223557 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:59.224786 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:59.226569 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:59.227359 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:59.227501 (+ 142us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:13:59.232405 (+ 4904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:13:59.232412 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:13:59.232426 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:13:59.232458 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:13:59.235596 (+ 3138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:13:59.236376 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:13:59.236390 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:13:59.236395 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:13:59.236540 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:13:59.237014 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:13:59.237018 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:13:59.237020 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:13:59.237577 (+ 557us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:13:59.237924 (+ 347us) server_negotiation.cc:1035] Waiting for connection context
0926 15:13:59.238392 (+ 468us) server_negotiation.cc:299] Negotiation successful
0926 15:13:59.238669 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":556,"thread_start_us":364,"threads_started":1}
I20260926 15:13:59.240840 427 raft_consensus.cc:348] T 00000000000000000000000000000000 P c86a16ef669d4827a4e269f15d1af196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c86a16ef669d4827a4e269f15d1af196" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37523 } }
I20260926 15:13:59.241200 427 raft_consensus.cc:374] T 00000000000000000000000000000000 P c86a16ef669d4827a4e269f15d1af196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:13:59.241326 427 raft_consensus.cc:729] T 00000000000000000000000000000000 P c86a16ef669d4827a4e269f15d1af196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c86a16ef669d4827a4e269f15d1af196, State: Initialized, Role: FOLLOWER
I20260926 15:13:59.241890 427 consensus_queue.cc:260] T 00000000000000000000000000000000 P c86a16ef669d4827a4e269f15d1af196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c86a16ef669d4827a4e269f15d1af196" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37523 } }
I20260926 15:13:59.242156 427 raft_consensus.cc:388] T 00000000000000000000000000000000 P c86a16ef669d4827a4e269f15d1af196 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 15:13:59.242302 427 raft_consensus.cc:482] T 00000000000000000000000000000000 P c86a16ef669d4827a4e269f15d1af196 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 15:13:59.242475 427 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c86a16ef669d4827a4e269f15d1af196 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:13:59.243754 12427 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 15:13:59.245528 427 raft_consensus.cc:504] T 00000000000000000000000000000000 P c86a16ef669d4827a4e269f15d1af196 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c86a16ef669d4827a4e269f15d1af196" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37523 } }
I20260926 15:13:59.246070 427 leader_election.cc:302] T 00000000000000000000000000000000 P c86a16ef669d4827a4e269f15d1af196 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c86a16ef669d4827a4e269f15d1af196; no voters:
I20260926 15:13:59.246764 427 leader_election.cc:288] T 00000000000000000000000000000000 P c86a16ef669d4827a4e269f15d1af196 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 15:13:59.246891 432 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c86a16ef669d4827a4e269f15d1af196 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:13:59.247814 432 raft_consensus.cc:686] T 00000000000000000000000000000000 P c86a16ef669d4827a4e269f15d1af196 [term 1 LEADER]: Becoming Leader. State: Replica: c86a16ef669d4827a4e269f15d1af196, State: Running, Role: LEADER
I20260926 15:13:59.248471 432 consensus_queue.cc:237] T 00000000000000000000000000000000 P c86a16ef669d4827a4e269f15d1af196 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c86a16ef669d4827a4e269f15d1af196" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37523 } }
I20260926 15:13:59.251276 435 sys_catalog.cc:455] T 00000000000000000000000000000000 P c86a16ef669d4827a4e269f15d1af196 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c86a16ef669d4827a4e269f15d1af196" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c86a16ef669d4827a4e269f15d1af196" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37523 } } }
I20260926 15:13:59.252030 435 sys_catalog.cc:458] T 00000000000000000000000000000000 P c86a16ef669d4827a4e269f15d1af196 [sys.catalog]: This master's current role is: LEADER
I20260926 15:13:59.251932 433 sys_catalog.cc:455] T 00000000000000000000000000000000 P c86a16ef669d4827a4e269f15d1af196 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c86a16ef669d4827a4e269f15d1af196. Latest consensus state: current_term: 1 leader_uuid: "c86a16ef669d4827a4e269f15d1af196" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c86a16ef669d4827a4e269f15d1af196" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37523 } } }
I20260926 15:13:59.252267 427 sys_catalog.cc:564] T 00000000000000000000000000000000 P c86a16ef669d4827a4e269f15d1af196 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:13:59.252391 433 sys_catalog.cc:458] T 00000000000000000000000000000000 P c86a16ef669d4827a4e269f15d1af196 [sys.catalog]: This master's current role is: LEADER
I20260926 15:13:59.253564 440 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:13:59.263440 440 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 15:13:59.278616 430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:13:59.254890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44777 (local address 127.0.0.1:37523)
0926 15:13:59.255116 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0926 15:13:59.255124 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 15:13:59.259998 (+ 4874us) server_negotiation.cc:407] Connection header received
0926 15:13:59.260292 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:13:59.260297 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:13:59.260376 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:13:59.260454 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:13:59.262584 (+ 2130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:59.263741 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:59.265565 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:13:59.266308 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:13:59.266429 (+ 121us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:13:59.269770 (+ 3341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:13:59.269776 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:13:59.269783 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:13:59.269822 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:13:59.274063 (+ 4241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:13:59.275309 (+ 1246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:13:59.275317 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:13:59.275324 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:13:59.275499 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:13:59.276171 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:13:59.276179 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:13:59.276185 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:13:59.276659 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:13:59.276964 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0926 15:13:59.277839 (+ 875us) server_negotiation.cc:299] Negotiation successful
0926 15:13:59.278077 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 15:13:59.284262 440 catalog_manager.cc:1347] Generated new cluster ID: 9696a25828024fb098e9735b388a35c7
I20260926 15:13:59.284392 440 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 15:13:59.305164 440 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 15:13:59.306438 440 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:13:59.314055 440 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c86a16ef669d4827a4e269f15d1af196: Generated new TSK 0
I20260926 15:13:59.314941 440 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 15:13:59.322053 12427 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 360
I20260926 15:13:59.323365 422 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:13:59.413702 12427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8uoJGv/build/asan/bin/kudu with pid 360
W20260926 15:13:59.432252 345 connection.cc:535] client connection to 127.0.0.1:37523 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37523): Connection reset by peer (error 104)
2026-09-26T15:13:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2001 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790435318278681-12427-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 15:13:59 dist-test-slave-865v krb5kdc[459](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 15:13:59 dist-test-slave-865v krb5kdc[459](info): set up 2 sockets
Sep 26 15:13:59 dist-test-slave-865v krb5kdc[459](info): commencing operation
krb5kdc: starting...
W20260926 15:14:00.517951 12427 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790435318278681-12427-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790435318278681-12427-0/krb5kdc/test-user.keytab.
Sep 26 15:14:00 dist-test-slave-865v krb5kdc[459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435640, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T15:14:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T15:14:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790435318278681-12427-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790435318278681-12427-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 15:14:00.769219 12427 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8uoJGv/build/asan/bin/kudu
/tmp/dist-test-task8uoJGv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790435318278681-12427-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790435318278681-12427-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790435318278681-12427-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39383
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790435318278681-12427-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39383
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790435318278681-12427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790435318278681-12427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790435318278681-12427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:14:01.094247 472 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:14:01.094556 472 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:14:01.094785 472 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:14:01.102917 472 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:14:01.103034 472 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:14:01.103128 472 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:14:01.103222 472 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:14:01.103313 472 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:14:01.113186 472 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790435318278681-12427-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790435318278681-12427-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39383
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790435318278681-12427-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39383
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790435318278681-12427-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:14:01.114477 472 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:14:01.115718 472 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:14:01.128330 478 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:14:01.128286 480 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:14:01.128547 472 server_base.cc:1034] running on GCE node
W20260926 15:14:01.128868 477 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:14:01.129788 472 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:14:01.131196 472 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:14:01.132589 472 hybrid_clock.cc:648] HybridClock initialized: now 1790435641132477 us; error 50 us; skew 500 ppm
I20260926 15:14:01.133225 472 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:14:01 dist-test-slave-865v krb5kdc[459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435641, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:14:01.137172 472 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 15:14:01.139156 472 webserver.cc:460] Webserver started at http://127.0.0.1:41607/ using document root <none> and password file <none>
I20260926 15:14:01.139987 472 fs_manager.cc:359] Metadata directory not provided
I20260926 15:14:01.140133 472 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:14:01.140460 472 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:14:01.143721 472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790435318278681-12427-0/minicluster-data/master-0/data/instance:
uuid: "cb7a37e353f0476182f69d9825dac96b"
format_stamp: "Formatted at 2026-09-26 15:14:01 on dist-test-slave-865v"
server_key: "3c0617716841f7d698b88ff10843ca8e"
server_key_iv: "8687506c5b2345eecc21bc284d49e7fc"
server_key_version: "encryptionkey@0"
I20260926 15:14:01.144518 472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790435318278681-12427-0/minicluster-data/master-0/wal/instance:
uuid: "cb7a37e353f0476182f69d9825dac96b"
format_stamp: "Formatted at 2026-09-26 15:14:01 on dist-test-slave-865v"
server_key: "3c0617716841f7d698b88ff10843ca8e"
server_key_iv: "8687506c5b2345eecc21bc284d49e7fc"
server_key_version: "encryptionkey@0"
I20260926 15:14:01.150538 472 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260926 15:14:01.155000 487 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:14:01.156721 472 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260926 15:14:01.156961 472 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790435318278681-12427-0/minicluster-data/master-0/data,/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790435318278681-12427-0/minicluster-data/master-0/wal
uuid: "cb7a37e353f0476182f69d9825dac96b"
format_stamp: "Formatted at 2026-09-26 15:14:01 on dist-test-slave-865v"
server_key: "3c0617716841f7d698b88ff10843ca8e"
server_key_iv: "8687506c5b2345eecc21bc284d49e7fc"
server_key_version: "encryptionkey@0"
I20260926 15:14:01.157236 472 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790435318278681-12427-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790435318278681-12427-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790435318278681-12427-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:14:01.179301 472 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:14:01.184424 472 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:14:01.184742 472 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:14:01.206156 472 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39383
I20260926 15:14:01.206207 538 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39383 every 8 connection(s)
I20260926 15:14:01.207544 472 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
I20260926 15:14:01.211856 539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:14:01.214238 12427 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8uoJGv/build/asan/bin/kudu as pid 472
I20260926 15:14:01.214424 12427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790435318278681-12427-0/minicluster-data/master-0/wal/instance
I20260926 15:14:01.214882 12427 external_mini_cluster.cc:1428] Setting key 162c3d5b426bddfcb292a5db2269e0a4
I20260926 15:14:01.225108 539 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb7a37e353f0476182f69d9825dac96b: Bootstrap starting.
I20260926 15:14:01.228899 539 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cb7a37e353f0476182f69d9825dac96b: Neither blocks nor log segments found. Creating new log.
I20260926 15:14:01.230387 539 log.cc:824] T 00000000000000000000000000000000 P cb7a37e353f0476182f69d9825dac96b: Log is configured to *not* fsync() on all Append() calls
I20260926 15:14:01.230616 542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:14:01.216686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56509 (local address 127.0.0.1:39383)
0926 15:14:01.217559 (+ 873us) server_negotiation.cc:206] Beginning negotiation
0926 15:14:01.217574 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 15:14:01.217626 (+ 52us) server_negotiation.cc:407] Connection header received
0926 15:14:01.220008 (+ 2382us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:14:01.220036 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:14:01.220533 (+ 497us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:14:01.220941 (+ 408us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:14:01.221477 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:14:01.223490 (+ 2013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:14:01.224783 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:14:01.225791 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:14:01.225963 (+ 172us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:14:01.227640 (+ 1677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:14:01.227652 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:14:01.227671 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:14:01.228364 (+ 693us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:14:01.228462 (+ 98us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:14:01.228497 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:14:01.228656 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0926 15:14:01.229009 (+ 353us) server_negotiation.cc:299] Negotiation successful
0926 15:14:01.229203 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":632,"thread_start_us":384,"threads_started":1}
I20260926 15:14:01.233476 539 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb7a37e353f0476182f69d9825dac96b: No bootstrap required, opened a new log
I20260926 15:14:01.234594 12427 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 15:14:01.242255 539 raft_consensus.cc:348] T 00000000000000000000000000000000 P cb7a37e353f0476182f69d9825dac96b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb7a37e353f0476182f69d9825dac96b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39383 } }
I20260926 15:14:01.242633 539 raft_consensus.cc:374] T 00000000000000000000000000000000 P cb7a37e353f0476182f69d9825dac96b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:14:01.242750 539 raft_consensus.cc:729] T 00000000000000000000000000000000 P cb7a37e353f0476182f69d9825dac96b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb7a37e353f0476182f69d9825dac96b, State: Initialized, Role: FOLLOWER
I20260926 15:14:01.243464 539 consensus_queue.cc:260] T 00000000000000000000000000000000 P cb7a37e353f0476182f69d9825dac96b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb7a37e353f0476182f69d9825dac96b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39383 } }
I20260926 15:14:01.243752 539 raft_consensus.cc:388] T 00000000000000000000000000000000 P cb7a37e353f0476182f69d9825dac96b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 15:14:01.243922 539 raft_consensus.cc:482] T 00000000000000000000000000000000 P cb7a37e353f0476182f69d9825dac96b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 15:14:01.244112 539 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cb7a37e353f0476182f69d9825dac96b [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:14:01.245829 542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:14:01.239087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58267 (local address 127.0.0.1:39383)
0926 15:14:01.239317 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0926 15:14:01.239323 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:14:01.239345 (+ 22us) server_negotiation.cc:407] Connection header received
0926 15:14:01.239525 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:14:01.239530 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:14:01.239603 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:14:01.239704 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:14:01.240287 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:14:01.241503 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:14:01.242960 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:14:01.243602 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:14:01.243727 (+ 125us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:14:01.244956 (+ 1229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:14:01.244964 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:14:01.244973 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:14:01.245000 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:14:01.245067 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:14:01.245112 (+ 45us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:14:01.245232 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0926 15:14:01.245438 (+ 206us) server_negotiation.cc:299] Negotiation successful
0926 15:14:01.245497 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 15:14:01.246996 539 raft_consensus.cc:504] T 00000000000000000000000000000000 P cb7a37e353f0476182f69d9825dac96b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb7a37e353f0476182f69d9825dac96b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39383 } }
I20260926 15:14:01.247543 539 leader_election.cc:302] T 00000000000000000000000000000000 P cb7a37e353f0476182f69d9825dac96b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cb7a37e353f0476182f69d9825dac96b; no voters:
I20260926 15:14:01.248145 539 leader_election.cc:288] T 00000000000000000000000000000000 P cb7a37e353f0476182f69d9825dac96b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 15:14:01.248309 549 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cb7a37e353f0476182f69d9825dac96b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:14:01.249045 549 raft_consensus.cc:686] T 00000000000000000000000000000000 P cb7a37e353f0476182f69d9825dac96b [term 1 LEADER]: Becoming Leader. State: Replica: cb7a37e353f0476182f69d9825dac96b, State: Running, Role: LEADER
I20260926 15:14:01.249909 549 consensus_queue.cc:237] T 00000000000000000000000000000000 P cb7a37e353f0476182f69d9825dac96b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb7a37e353f0476182f69d9825dac96b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39383 } }
I20260926 15:14:01.250485 539 sys_catalog.cc:564] T 00000000000000000000000000000000 P cb7a37e353f0476182f69d9825dac96b [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:14:01.253299 551 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb7a37e353f0476182f69d9825dac96b [sys.catalog]: SysCatalogTable state changed. Reason: New leader cb7a37e353f0476182f69d9825dac96b. Latest consensus state: current_term: 1 leader_uuid: "cb7a37e353f0476182f69d9825dac96b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb7a37e353f0476182f69d9825dac96b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39383 } } }
I20260926 15:14:01.253638 551 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb7a37e353f0476182f69d9825dac96b [sys.catalog]: This master's current role is: LEADER
I20260926 15:14:01.254344 550 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb7a37e353f0476182f69d9825dac96b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cb7a37e353f0476182f69d9825dac96b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb7a37e353f0476182f69d9825dac96b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39383 } } }
I20260926 15:14:01.254794 550 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb7a37e353f0476182f69d9825dac96b [sys.catalog]: This master's current role is: LEADER
I20260926 15:14:01.255993 557 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:14:01.262818 557 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 15:14:01.272399 557 catalog_manager.cc:1347] Generated new cluster ID: c0a6a201d885452da7e1d2151090792f
I20260926 15:14:01.272562 557 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 15:14:01.286309 557 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 15:14:01.288076 557 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:14:01.306105 557 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cb7a37e353f0476182f69d9825dac96b: Generated new TSK 0
I20260926 15:14:01.307039 557 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 15:14:01.321179 12427 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 472
W20260926 15:14:01.322445 534 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:56509
W20260926 15:14:01.323271 12427 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 472: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 15:14:01.323477 12427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8uoJGv/build/asan/bin/kudu with pid 472
W20260926 15:14:01.340298 457 connection.cc:535] client connection to 127.0.0.1:39383 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39383): Connection reset by peer (error 104)
2026-09-26T15:14:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1907 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790435318278681-12427-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 15:14:01 dist-test-slave-865v krb5kdc[570](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 15:14:01 dist-test-slave-865v krb5kdc[570](info): set up 2 sockets
Sep 26 15:14:01 dist-test-slave-865v krb5kdc[570](info): commencing operation
krb5kdc: starting...
W20260926 15:14:02.473701 12427 mini_kdc.cc:121] Time spent starting KDC: real 1.115s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790435318278681-12427-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790435318278681-12427-0/krb5kdc/test-user.keytab.
Sep 26 15:14:02 dist-test-slave-865v krb5kdc[570](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435642, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T15:14:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T15:14:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790435318278681-12427-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790435318278681-12427-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 15:14:02.764907 12427 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8uoJGv/build/asan/bin/kudu
/tmp/dist-test-task8uoJGv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790435318278681-12427-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790435318278681-12427-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790435318278681-12427-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38905
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790435318278681-12427-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38905
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790435318278681-12427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790435318278681-12427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790435318278681-12427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:14:03.097347 583 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:14:03.097695 583 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:14:03.097858 583 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:14:03.105250 583 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:14:03.105392 583 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:14:03.105494 583 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:14:03.105599 583 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:14:03.105695 583 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:14:03.115543 583 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790435318278681-12427-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790435318278681-12427-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38905
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790435318278681-12427-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38905
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790435318278681-12427-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:14:03.116845 583 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:14:03.118173 583 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:14:03.130007 589 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:14:03.130049 588 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:14:03.130089 591 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:14:03.131179 583 server_base.cc:1034] running on GCE node
I20260926 15:14:03.132232 583 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:14:03.133705 583 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:14:03.134967 583 hybrid_clock.cc:648] HybridClock initialized: now 1790435643134938 us; error 48 us; skew 500 ppm
I20260926 15:14:03.135344 583 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:14:03 dist-test-slave-865v krb5kdc[570](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435643, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:14:03.138780 583 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 15:14:03.140345 583 webserver.cc:460] Webserver started at http://127.0.0.1:34821/ using document root <none> and password file <none>
I20260926 15:14:03.141040 583 fs_manager.cc:359] Metadata directory not provided
I20260926 15:14:03.141259 583 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:14:03.141571 583 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:14:03.145023 583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790435318278681-12427-0/minicluster-data/master-0/data/instance:
uuid: "4cb72e5119154a8d8887c438893677fa"
format_stamp: "Formatted at 2026-09-26 15:14:03 on dist-test-slave-865v"
server_key: "abf80999759dd986e41ce20a3f6f564c"
server_key_iv: "b0457c5fd330d07abbc7aab7d4948558"
server_key_version: "encryptionkey@0"
I20260926 15:14:03.146164 583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790435318278681-12427-0/minicluster-data/master-0/wal/instance:
uuid: "4cb72e5119154a8d8887c438893677fa"
format_stamp: "Formatted at 2026-09-26 15:14:03 on dist-test-slave-865v"
server_key: "abf80999759dd986e41ce20a3f6f564c"
server_key_iv: "b0457c5fd330d07abbc7aab7d4948558"
server_key_version: "encryptionkey@0"
I20260926 15:14:03.151611 583 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 15:14:03.155315 598 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:14:03.156955 583 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 15:14:03.157198 583 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790435318278681-12427-0/minicluster-data/master-0/data,/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790435318278681-12427-0/minicluster-data/master-0/wal
uuid: "4cb72e5119154a8d8887c438893677fa"
format_stamp: "Formatted at 2026-09-26 15:14:03 on dist-test-slave-865v"
server_key: "abf80999759dd986e41ce20a3f6f564c"
server_key_iv: "b0457c5fd330d07abbc7aab7d4948558"
server_key_version: "encryptionkey@0"
I20260926 15:14:03.157413 583 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790435318278681-12427-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790435318278681-12427-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790435318278681-12427-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:14:03.163514 583 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:14:03.167812 583 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:14:03.168141 583 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:14:03.188796 583 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38905
I20260926 15:14:03.188843 649 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38905 every 8 connection(s)
I20260926 15:14:03.190362 583 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
I20260926 15:14:03.194257 12427 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8uoJGv/build/asan/bin/kudu as pid 583
I20260926 15:14:03.194362 650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:14:03.194463 12427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790435318278681-12427-0/minicluster-data/master-0/wal/instance
I20260926 15:14:03.195014 12427 external_mini_cluster.cc:1428] Setting key 81d223b35fb7f3acce36c82015457c66
I20260926 15:14:03.204008 653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:14:03.197374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44539 (local address 127.0.0.1:38905)
0926 15:14:03.198201 (+ 827us) server_negotiation.cc:206] Beginning negotiation
0926 15:14:03.198212 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 15:14:03.198250 (+ 38us) server_negotiation.cc:407] Connection header received
0926 15:14:03.200582 (+ 2332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:14:03.200604 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:14:03.200946 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:14:03.201365 (+ 419us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:14:03.202370 (+ 1005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:14:03.202377 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:14:03.202389 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:14:03.202406 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:14:03.202455 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:14:03.202473 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:14:03.202638 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0926 15:14:03.202803 (+ 165us) server_negotiation.cc:299] Negotiation successful
0926 15:14:03.203018 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":597,"thread_start_us":393,"threads_started":1}
I20260926 15:14:03.207994 12427 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 15:14:03.208297 650 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4cb72e5119154a8d8887c438893677fa: Bootstrap starting.
I20260926 15:14:03.211288 650 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4cb72e5119154a8d8887c438893677fa: Neither blocks nor log segments found. Creating new log.
I20260926 15:14:03.212908 650 log.cc:824] T 00000000000000000000000000000000 P 4cb72e5119154a8d8887c438893677fa: Log is configured to *not* fsync() on all Append() calls
I20260926 15:14:03.215232 653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:14:03.212674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55865 (local address 127.0.0.1:38905)
0926 15:14:03.212857 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0926 15:14:03.212861 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:14:03.213050 (+ 189us) server_negotiation.cc:407] Connection header received
0926 15:14:03.213230 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:14:03.213234 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:14:03.213280 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:14:03.213359 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:14:03.214513 (+ 1154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:14:03.214518 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:14:03.214521 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:14:03.214537 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:14:03.214577 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:14:03.214595 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:14:03.214717 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0926 15:14:03.214846 (+ 129us) server_negotiation.cc:299] Negotiation successful
0926 15:14:03.214895 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 15:14:03.216264 650 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4cb72e5119154a8d8887c438893677fa: No bootstrap required, opened a new log
I20260926 15:14:03.224431 650 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4cb72e5119154a8d8887c438893677fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cb72e5119154a8d8887c438893677fa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38905 } }
I20260926 15:14:03.225195 650 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4cb72e5119154a8d8887c438893677fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:14:03.225342 650 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4cb72e5119154a8d8887c438893677fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cb72e5119154a8d8887c438893677fa, State: Initialized, Role: FOLLOWER
I20260926 15:14:03.225975 650 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4cb72e5119154a8d8887c438893677fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cb72e5119154a8d8887c438893677fa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38905 } }
I20260926 15:14:03.226215 650 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4cb72e5119154a8d8887c438893677fa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 15:14:03.226363 650 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4cb72e5119154a8d8887c438893677fa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 15:14:03.226543 650 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4cb72e5119154a8d8887c438893677fa [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:14:03.229015 650 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4cb72e5119154a8d8887c438893677fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cb72e5119154a8d8887c438893677fa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38905 } }
I20260926 15:14:03.229568 650 leader_election.cc:302] T 00000000000000000000000000000000 P 4cb72e5119154a8d8887c438893677fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4cb72e5119154a8d8887c438893677fa; no voters:
I20260926 15:14:03.230197 650 leader_election.cc:288] T 00000000000000000000000000000000 P 4cb72e5119154a8d8887c438893677fa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 15:14:03.230376 660 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4cb72e5119154a8d8887c438893677fa [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:14:03.231532 660 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4cb72e5119154a8d8887c438893677fa [term 1 LEADER]: Becoming Leader. State: Replica: 4cb72e5119154a8d8887c438893677fa, State: Running, Role: LEADER
I20260926 15:14:03.232331 660 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4cb72e5119154a8d8887c438893677fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cb72e5119154a8d8887c438893677fa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38905 } }
I20260926 15:14:03.232501 650 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4cb72e5119154a8d8887c438893677fa [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:14:03.235245 662 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4cb72e5119154a8d8887c438893677fa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4cb72e5119154a8d8887c438893677fa. Latest consensus state: current_term: 1 leader_uuid: "4cb72e5119154a8d8887c438893677fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cb72e5119154a8d8887c438893677fa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38905 } } }
I20260926 15:14:03.235710 662 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4cb72e5119154a8d8887c438893677fa [sys.catalog]: This master's current role is: LEADER
I20260926 15:14:03.235721 661 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4cb72e5119154a8d8887c438893677fa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4cb72e5119154a8d8887c438893677fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cb72e5119154a8d8887c438893677fa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38905 } } }
I20260926 15:14:03.236332 661 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4cb72e5119154a8d8887c438893677fa [sys.catalog]: This master's current role is: LEADER
I20260926 15:14:03.236793 669 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:14:03.242595 669 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 15:14:03.253615 669 catalog_manager.cc:1347] Generated new cluster ID: 9086da9f06454872803e6b56daee8e42
I20260926 15:14:03.253732 669 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 15:14:03.279100 669 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 15:14:03.279557 669 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:14:03.290252 669 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4cb72e5119154a8d8887c438893677fa: Generated new TSK 0
I20260926 15:14:03.291132 669 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 15:14:03.299506 12427 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 583
W20260926 15:14:03.300572 645 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:44539
W20260926 15:14:03.301275 12427 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 583: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 15:14:03.301429 12427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8uoJGv/build/asan/bin/kudu with pid 583
2026-09-26T15:14:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1975 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790435318278681-12427-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 15:14:03 dist-test-slave-865v krb5kdc[681](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 15:14:03 dist-test-slave-865v krb5kdc[681](info): set up 2 sockets
Sep 26 15:14:03 dist-test-slave-865v krb5kdc[681](info): commencing operation
krb5kdc: starting...
W20260926 15:14:04.409595 12427 mini_kdc.cc:121] Time spent starting KDC: real 1.075s user 0.004s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790435318278681-12427-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790435318278681-12427-0/krb5kdc/test-user.keytab.
Sep 26 15:14:04 dist-test-slave-865v krb5kdc[681](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435644, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T15:14:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T15:14:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790435318278681-12427-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790435318278681-12427-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 15:14:04.672729 12427 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8uoJGv/build/asan/bin/kudu
/tmp/dist-test-task8uoJGv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790435318278681-12427-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790435318278681-12427-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790435318278681-12427-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41433
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790435318278681-12427-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41433
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790435318278681-12427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790435318278681-12427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790435318278681-12427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:14:05.015571 694 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:14:05.015887 694 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:14:05.015991 694 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:14:05.023972 694 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:14:05.024117 694 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:14:05.024165 694 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:14:05.024219 694 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:14:05.024264 694 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:14:05.034317 694 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790435318278681-12427-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790435318278681-12427-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41433
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790435318278681-12427-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41433
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790435318278681-12427-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:14:05.035578 694 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:14:05.036799 694 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:14:05.048967 699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:14:05.048933 702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:14:05.048893 700 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:14:05.050076 694 server_base.cc:1034] running on GCE node
I20260926 15:14:05.050872 694 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:14:05.052394 694 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:14:05.053661 694 hybrid_clock.cc:648] HybridClock initialized: now 1790435645053611 us; error 53 us; skew 500 ppm
I20260926 15:14:05.054054 694 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:14:05 dist-test-slave-865v krb5kdc[681](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435645, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:14:05.062219 694 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 15:14:05.064190 694 webserver.cc:460] Webserver started at http://127.0.0.1:34423/ using document root <none> and password file <none>
I20260926 15:14:05.065907 694 fs_manager.cc:359] Metadata directory not provided
I20260926 15:14:05.066095 694 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:14:05.066442 694 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:14:05.069600 694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790435318278681-12427-0/minicluster-data/master-0/data/instance:
uuid: "116c04c053d34295992a67b2d9a00ce2"
format_stamp: "Formatted at 2026-09-26 15:14:05 on dist-test-slave-865v"
server_key: "e9409a0fd31de5520399aee3ca544a4e"
server_key_iv: "41261dc1d3bffab66f7b979fd627c79f"
server_key_version: "encryptionkey@0"
I20260926 15:14:05.070395 694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790435318278681-12427-0/minicluster-data/master-0/wal/instance:
uuid: "116c04c053d34295992a67b2d9a00ce2"
format_stamp: "Formatted at 2026-09-26 15:14:05 on dist-test-slave-865v"
server_key: "e9409a0fd31de5520399aee3ca544a4e"
server_key_iv: "41261dc1d3bffab66f7b979fd627c79f"
server_key_version: "encryptionkey@0"
I20260926 15:14:05.075627 694 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 15:14:05.079336 709 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:14:05.080884 694 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 15:14:05.081211 694 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790435318278681-12427-0/minicluster-data/master-0/data,/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790435318278681-12427-0/minicluster-data/master-0/wal
uuid: "116c04c053d34295992a67b2d9a00ce2"
format_stamp: "Formatted at 2026-09-26 15:14:05 on dist-test-slave-865v"
server_key: "e9409a0fd31de5520399aee3ca544a4e"
server_key_iv: "41261dc1d3bffab66f7b979fd627c79f"
server_key_version: "encryptionkey@0"
I20260926 15:14:05.081434 694 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790435318278681-12427-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790435318278681-12427-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790435318278681-12427-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:14:05.098731 694 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:14:05.105942 694 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:14:05.106304 694 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:14:05.129602 694 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41433
I20260926 15:14:05.129622 760 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41433 every 8 connection(s)
I20260926 15:14:05.131245 694 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
I20260926 15:14:05.136991 761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:14:05.138903 12427 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8uoJGv/build/asan/bin/kudu as pid 694
I20260926 15:14:05.139094 12427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790435318278681-12427-0/minicluster-data/master-0/wal/instance
I20260926 15:14:05.139550 12427 external_mini_cluster.cc:1428] Setting key c36ab025f937cf7829b384c9e07e6064
I20260926 15:14:05.151772 761 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 116c04c053d34295992a67b2d9a00ce2: Bootstrap starting.
Sep 26 15:14:05 dist-test-slave-865v krb5kdc[681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435644, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 15:14:05.156067 761 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 116c04c053d34295992a67b2d9a00ce2: Neither blocks nor log segments found. Creating new log.
I20260926 15:14:05.157410 761 log.cc:824] T 00000000000000000000000000000000 P 116c04c053d34295992a67b2d9a00ce2: Log is configured to *not* fsync() on all Append() calls
I20260926 15:14:05.160810 761 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 116c04c053d34295992a67b2d9a00ce2: No bootstrap required, opened a new log
I20260926 15:14:05.165483 764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:14:05.141548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53941 (local address 127.0.0.1:41433)
0926 15:14:05.142378 (+ 830us) server_negotiation.cc:206] Beginning negotiation
0926 15:14:05.142394 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 15:14:05.142444 (+ 50us) server_negotiation.cc:407] Connection header received
0926 15:14:05.144153 (+ 1709us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:14:05.144186 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:14:05.144723 (+ 537us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:14:05.145187 (+ 464us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:14:05.146710 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:14:05.148755 (+ 2045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:14:05.150153 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:14:05.151136 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:14:05.151330 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:14:05.156711 (+ 5381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:14:05.156722 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:14:05.156739 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:14:05.156783 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:14:05.160744 (+ 3961us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:14:05.161633 (+ 889us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:14:05.161640 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:14:05.161648 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:14:05.161841 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:14:05.162265 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:14:05.162273 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:14:05.162279 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:14:05.162992 (+ 713us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:14:05.163332 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0926 15:14:05.163648 (+ 316us) server_negotiation.cc:299] Negotiation successful
0926 15:14:05.164107 (+ 459us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":571,"thread_start_us":353,"threads_started":1}
I20260926 15:14:05.170066 12427 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 15:14:05.170814 761 raft_consensus.cc:348] T 00000000000000000000000000000000 P 116c04c053d34295992a67b2d9a00ce2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "116c04c053d34295992a67b2d9a00ce2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41433 } }
I20260926 15:14:05.171164 761 raft_consensus.cc:374] T 00000000000000000000000000000000 P 116c04c053d34295992a67b2d9a00ce2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:14:05.171260 761 raft_consensus.cc:729] T 00000000000000000000000000000000 P 116c04c053d34295992a67b2d9a00ce2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 116c04c053d34295992a67b2d9a00ce2, State: Initialized, Role: FOLLOWER
I20260926 15:14:05.171820 761 consensus_queue.cc:260] T 00000000000000000000000000000000 P 116c04c053d34295992a67b2d9a00ce2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "116c04c053d34295992a67b2d9a00ce2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41433 } }
I20260926 15:14:05.172052 761 raft_consensus.cc:388] T 00000000000000000000000000000000 P 116c04c053d34295992a67b2d9a00ce2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 15:14:05.172212 761 raft_consensus.cc:482] T 00000000000000000000000000000000 P 116c04c053d34295992a67b2d9a00ce2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 15:14:05.172375 761 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 116c04c053d34295992a67b2d9a00ce2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:14:05.174803 761 raft_consensus.cc:504] T 00000000000000000000000000000000 P 116c04c053d34295992a67b2d9a00ce2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "116c04c053d34295992a67b2d9a00ce2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41433 } }
I20260926 15:14:05.175395 761 leader_election.cc:302] T 00000000000000000000000000000000 P 116c04c053d34295992a67b2d9a00ce2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 116c04c053d34295992a67b2d9a00ce2; no voters:
I20260926 15:14:05.176132 761 leader_election.cc:288] T 00000000000000000000000000000000 P 116c04c053d34295992a67b2d9a00ce2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 15:14:05.176283 771 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 116c04c053d34295992a67b2d9a00ce2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:14:05.176980 771 raft_consensus.cc:686] T 00000000000000000000000000000000 P 116c04c053d34295992a67b2d9a00ce2 [term 1 LEADER]: Becoming Leader. State: Replica: 116c04c053d34295992a67b2d9a00ce2, State: Running, Role: LEADER
I20260926 15:14:05.177739 771 consensus_queue.cc:237] T 00000000000000000000000000000000 P 116c04c053d34295992a67b2d9a00ce2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "116c04c053d34295992a67b2d9a00ce2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41433 } }
I20260926 15:14:05.179834 771 sys_catalog.cc:455] T 00000000000000000000000000000000 P 116c04c053d34295992a67b2d9a00ce2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 116c04c053d34295992a67b2d9a00ce2. Latest consensus state: current_term: 1 leader_uuid: "116c04c053d34295992a67b2d9a00ce2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "116c04c053d34295992a67b2d9a00ce2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41433 } } }
I20260926 15:14:05.181057 773 sys_catalog.cc:455] T 00000000000000000000000000000000 P 116c04c053d34295992a67b2d9a00ce2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "116c04c053d34295992a67b2d9a00ce2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "116c04c053d34295992a67b2d9a00ce2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41433 } } }
I20260926 15:14:05.181836 773 sys_catalog.cc:458] T 00000000000000000000000000000000 P 116c04c053d34295992a67b2d9a00ce2 [sys.catalog]: This master's current role is: LEADER
I20260926 15:14:05.181190 771 sys_catalog.cc:458] T 00000000000000000000000000000000 P 116c04c053d34295992a67b2d9a00ce2 [sys.catalog]: This master's current role is: LEADER
I20260926 15:14:05.184581 761 sys_catalog.cc:564] T 00000000000000000000000000000000 P 116c04c053d34295992a67b2d9a00ce2 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:14:05.187803 775 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:14:05.193820 775 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 15:14:05.195815 764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:14:05.175535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38227 (local address 127.0.0.1:41433)
0926 15:14:05.175729 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0926 15:14:05.175737 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 15:14:05.176341 (+ 604us) server_negotiation.cc:407] Connection header received
0926 15:14:05.176593 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:14:05.176598 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:14:05.176694 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:14:05.176851 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:14:05.179113 (+ 2262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:14:05.180960 (+ 1847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:14:05.183306 (+ 2346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:14:05.184017 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:14:05.184161 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:14:05.187504 (+ 3343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:14:05.187512 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:14:05.187516 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:14:05.187553 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:14:05.191772 (+ 4219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:14:05.192733 (+ 961us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:14:05.192738 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:14:05.192741 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:14:05.192885 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:14:05.193386 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:14:05.193391 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:14:05.193395 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:14:05.193846 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:14:05.195054 (+ 1208us) server_negotiation.cc:1035] Waiting for connection context
0926 15:14:05.195219 (+ 165us) server_negotiation.cc:299] Negotiation successful
0926 15:14:05.195434 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 15:14:05.203094 775 catalog_manager.cc:1347] Generated new cluster ID: 7e0099dca82641d1be37c8b7008337dd
I20260926 15:14:05.203256 775 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 15:14:05.217247 775 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 15:14:05.219080 775 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:14:05.240880 775 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 116c04c053d34295992a67b2d9a00ce2: Generated new TSK 0
I20260926 15:14:05.241796 775 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 15:14:05.253021 12427 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 694
I20260926 15:14:05.254355 756 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:14:05.346119 12427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8uoJGv/build/asan/bin/kudu with pid 694
2026-09-26T15:14:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2044 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790435318278681-12427-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 15:14:05 dist-test-slave-865v krb5kdc[793](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 15:14:05 dist-test-slave-865v krb5kdc[793](info): set up 2 sockets
Sep 26 15:14:05 dist-test-slave-865v krb5kdc[793](info): commencing operation
krb5kdc: starting...
W20260926 15:14:06.453524 12427 mini_kdc.cc:121] Time spent starting KDC: real 1.074s user 0.004s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790435318278681-12427-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790435318278681-12427-0/krb5kdc/test-user.keytab.
Sep 26 15:14:06 dist-test-slave-865v krb5kdc[793](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435646, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T15:14:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T15:14:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790435318278681-12427-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790435318278681-12427-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 15:14:06.716898 12427 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8uoJGv/build/asan/bin/kudu
/tmp/dist-test-task8uoJGv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790435318278681-12427-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790435318278681-12427-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790435318278681-12427-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44479
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790435318278681-12427-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44479
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790435318278681-12427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790435318278681-12427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790435318278681-12427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:14:07.064836 806 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:14:07.065207 806 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:14:07.065371 806 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:14:07.073388 806 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:14:07.073509 806 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:14:07.073591 806 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:14:07.073683 806 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:14:07.073769 806 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:14:07.083921 806 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790435318278681-12427-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790435318278681-12427-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44479
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790435318278681-12427-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44479
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790435318278681-12427-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:14:07.085211 806 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:14:07.086485 806 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:14:07.098773 811 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:14:07.098794 812 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:14:07.098837 814 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:14:07.100024 806 server_base.cc:1034] running on GCE node
I20260926 15:14:07.100785 806 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:14:07.102269 806 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:14:07.103494 806 hybrid_clock.cc:648] HybridClock initialized: now 1790435647103418 us; error 59 us; skew 500 ppm
I20260926 15:14:07.103816 806 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:14:07 dist-test-slave-865v krb5kdc[793](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435647, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:14:07.107264 806 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 15:14:07.108801 806 webserver.cc:460] Webserver started at http://127.0.0.1:46599/ using document root <none> and password file <none>
I20260926 15:14:07.109562 806 fs_manager.cc:359] Metadata directory not provided
I20260926 15:14:07.109689 806 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:14:07.110011 806 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:14:07.112892 806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790435318278681-12427-0/minicluster-data/master-0/data/instance:
uuid: "8636d8dbea8341d29032b0ddf2346de9"
format_stamp: "Formatted at 2026-09-26 15:14:07 on dist-test-slave-865v"
server_key: "9891c8863068687968424f388d9273a6"
server_key_iv: "97a10454bb98a93ff49a70be2879907d"
server_key_version: "encryptionkey@0"
I20260926 15:14:07.113730 806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790435318278681-12427-0/minicluster-data/master-0/wal/instance:
uuid: "8636d8dbea8341d29032b0ddf2346de9"
format_stamp: "Formatted at 2026-09-26 15:14:07 on dist-test-slave-865v"
server_key: "9891c8863068687968424f388d9273a6"
server_key_iv: "97a10454bb98a93ff49a70be2879907d"
server_key_version: "encryptionkey@0"
I20260926 15:14:07.118847 806 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 15:14:07.122418 821 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:14:07.123973 806 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260926 15:14:07.124176 806 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790435318278681-12427-0/minicluster-data/master-0/data,/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790435318278681-12427-0/minicluster-data/master-0/wal
uuid: "8636d8dbea8341d29032b0ddf2346de9"
format_stamp: "Formatted at 2026-09-26 15:14:07 on dist-test-slave-865v"
server_key: "9891c8863068687968424f388d9273a6"
server_key_iv: "97a10454bb98a93ff49a70be2879907d"
server_key_version: "encryptionkey@0"
I20260926 15:14:07.124392 806 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790435318278681-12427-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790435318278681-12427-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790435318278681-12427-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:14:07.143724 806 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:14:07.149312 806 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:14:07.149647 806 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:14:07.171241 806 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44479
I20260926 15:14:07.171479 872 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44479 every 8 connection(s)
I20260926 15:14:07.172654 806 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
I20260926 15:14:07.174160 12427 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8uoJGv/build/asan/bin/kudu as pid 806
I20260926 15:14:07.174341 12427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790435318278681-12427-0/minicluster-data/master-0/wal/instance
I20260926 15:14:07.174926 12427 external_mini_cluster.cc:1428] Setting key b2bbe2ac1a42425342686512a7b8598c
I20260926 15:14:07.177454 873 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:14:07.188380 876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:14:07.176734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46457 (local address 127.0.0.1:44479)
0926 15:14:07.177590 (+ 856us) server_negotiation.cc:206] Beginning negotiation
0926 15:14:07.177603 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 15:14:07.177642 (+ 39us) server_negotiation.cc:407] Connection header received
0926 15:14:07.179261 (+ 1619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:14:07.179281 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:14:07.179646 (+ 365us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:14:07.179976 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:14:07.180534 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:14:07.181934 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:14:07.183757 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:14:07.184422 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:14:07.184559 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:14:07.185906 (+ 1347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:14:07.185914 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:14:07.185928 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:14:07.186512 (+ 584us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:14:07.186603 (+ 91us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:14:07.186645 (+ 42us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:14:07.186736 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 15:14:07.187027 (+ 291us) server_negotiation.cc:299] Negotiation successful
0926 15:14:07.187303 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":608,"thread_start_us":412,"threads_started":1}
I20260926 15:14:07.190357 873 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8636d8dbea8341d29032b0ddf2346de9: Bootstrap starting.
I20260926 15:14:07.192790 12427 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 15:14:07.193809 873 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8636d8dbea8341d29032b0ddf2346de9: Neither blocks nor log segments found. Creating new log.
I20260926 15:14:07.194988 873 log.cc:824] T 00000000000000000000000000000000 P 8636d8dbea8341d29032b0ddf2346de9: Log is configured to *not* fsync() on all Append() calls
I20260926 15:14:07.198823 873 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8636d8dbea8341d29032b0ddf2346de9: No bootstrap required, opened a new log
I20260926 15:14:07.205184 876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:14:07.198494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52607 (local address 127.0.0.1:44479)
0926 15:14:07.198690 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0926 15:14:07.198697 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:14:07.199080 (+ 383us) server_negotiation.cc:407] Connection header received
0926 15:14:07.199241 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:14:07.199252 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:14:07.199344 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:14:07.199464 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:14:07.200008 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:14:07.201158 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:14:07.202444 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:14:07.203131 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:14:07.203251 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:14:07.204291 (+ 1040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:14:07.204297 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:14:07.204301 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:14:07.204323 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:14:07.204373 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:14:07.204393 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:14:07.204494 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0926 15:14:07.204695 (+ 201us) server_negotiation.cc:299] Negotiation successful
0926 15:14:07.204795 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 15:14:07.208461 873 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8636d8dbea8341d29032b0ddf2346de9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8636d8dbea8341d29032b0ddf2346de9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44479 } }
I20260926 15:14:07.208848 873 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8636d8dbea8341d29032b0ddf2346de9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:14:07.208969 873 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8636d8dbea8341d29032b0ddf2346de9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8636d8dbea8341d29032b0ddf2346de9, State: Initialized, Role: FOLLOWER
I20260926 15:14:07.209653 873 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8636d8dbea8341d29032b0ddf2346de9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8636d8dbea8341d29032b0ddf2346de9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44479 } }
I20260926 15:14:07.209867 873 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8636d8dbea8341d29032b0ddf2346de9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 15:14:07.210042 873 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8636d8dbea8341d29032b0ddf2346de9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 15:14:07.210229 873 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8636d8dbea8341d29032b0ddf2346de9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:14:07.213160 873 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8636d8dbea8341d29032b0ddf2346de9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8636d8dbea8341d29032b0ddf2346de9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44479 } }
I20260926 15:14:07.213703 873 leader_election.cc:302] T 00000000000000000000000000000000 P 8636d8dbea8341d29032b0ddf2346de9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8636d8dbea8341d29032b0ddf2346de9; no voters:
I20260926 15:14:07.214461 873 leader_election.cc:288] T 00000000000000000000000000000000 P 8636d8dbea8341d29032b0ddf2346de9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 15:14:07.214632 883 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8636d8dbea8341d29032b0ddf2346de9 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:14:07.215827 883 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8636d8dbea8341d29032b0ddf2346de9 [term 1 LEADER]: Becoming Leader. State: Replica: 8636d8dbea8341d29032b0ddf2346de9, State: Running, Role: LEADER
I20260926 15:14:07.216687 883 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8636d8dbea8341d29032b0ddf2346de9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8636d8dbea8341d29032b0ddf2346de9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44479 } }
I20260926 15:14:07.216917 873 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8636d8dbea8341d29032b0ddf2346de9 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:14:07.221784 885 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8636d8dbea8341d29032b0ddf2346de9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8636d8dbea8341d29032b0ddf2346de9. Latest consensus state: current_term: 1 leader_uuid: "8636d8dbea8341d29032b0ddf2346de9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8636d8dbea8341d29032b0ddf2346de9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44479 } } }
I20260926 15:14:07.221827 884 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8636d8dbea8341d29032b0ddf2346de9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8636d8dbea8341d29032b0ddf2346de9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8636d8dbea8341d29032b0ddf2346de9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44479 } } }
I20260926 15:14:07.222222 885 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8636d8dbea8341d29032b0ddf2346de9 [sys.catalog]: This master's current role is: LEADER
I20260926 15:14:07.222231 884 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8636d8dbea8341d29032b0ddf2346de9 [sys.catalog]: This master's current role is: LEADER
I20260926 15:14:07.224238 894 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:14:07.229229 894 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 15:14:07.241907 894 catalog_manager.cc:1347] Generated new cluster ID: f987a4c5831544a5ba7c696549ee529b
I20260926 15:14:07.242093 894 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 15:14:07.273892 894 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 15:14:07.275358 894 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:14:07.284359 894 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8636d8dbea8341d29032b0ddf2346de9: Generated new TSK 0
I20260926 15:14:07.285707 894 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 15:14:07.294150 12427 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 806
W20260926 15:14:07.295364 868 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:46457
W20260926 15:14:07.296152 12427 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 806: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 15:14:07.296311 12427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8uoJGv/build/asan/bin/kudu with pid 806
2026-09-26T15:14:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1953 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790435318278681-12427-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 15:14:07 dist-test-slave-865v krb5kdc[904](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 15:14:07 dist-test-slave-865v krb5kdc[904](info): set up 2 sockets
Sep 26 15:14:07 dist-test-slave-865v krb5kdc[904](info): commencing operation
krb5kdc: starting...
W20260926 15:14:08.433791 12427 mini_kdc.cc:121] Time spent starting KDC: real 1.101s user 0.002s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790435318278681-12427-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790435318278681-12427-0/krb5kdc/test-user.keytab.
Sep 26 15:14:08 dist-test-slave-865v krb5kdc[904](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435648, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T15:14:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T15:14:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790435318278681-12427-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790435318278681-12427-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 15:14:08.698294 12427 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8uoJGv/build/asan/bin/kudu
/tmp/dist-test-task8uoJGv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790435318278681-12427-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790435318278681-12427-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790435318278681-12427-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37429
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790435318278681-12427-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37429
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790435318278681-12427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790435318278681-12427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790435318278681-12427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:14:09.024235 917 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:14:09.024545 917 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:14:09.024703 917 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:14:09.031956 917 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:14:09.032063 917 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:14:09.032174 917 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:14:09.032292 917 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:14:09.032377 917 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:14:09.042125 917 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790435318278681-12427-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790435318278681-12427-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37429
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790435318278681-12427-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37429
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790435318278681-12427-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:14:09.043399 917 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:14:09.044639 917 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:14:09.056428 925 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:14:09.056473 923 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:14:09.056670 922 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:14:09.057804 917 server_base.cc:1034] running on GCE node
I20260926 15:14:09.058561 917 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:14:09.060125 917 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:14:09.061364 917 hybrid_clock.cc:648] HybridClock initialized: now 1790435649061321 us; error 65 us; skew 500 ppm
I20260926 15:14:09.061728 917 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:14:09 dist-test-slave-865v krb5kdc[904](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435649, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:14:09.065253 917 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 15:14:09.066709 917 webserver.cc:460] Webserver started at http://127.0.0.1:34543/ using document root <none> and password file <none>
I20260926 15:14:09.067368 917 fs_manager.cc:359] Metadata directory not provided
I20260926 15:14:09.067515 917 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:14:09.067808 917 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:14:09.071791 917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790435318278681-12427-0/minicluster-data/master-0/data/instance:
uuid: "1a93ab6102f94c8db0faad5f623b8486"
format_stamp: "Formatted at 2026-09-26 15:14:09 on dist-test-slave-865v"
server_key: "14560114da1fcc39969bd792ea094cab"
server_key_iv: "bd753c0b6dcf368cc37c3bcdadbc6e88"
server_key_version: "encryptionkey@0"
I20260926 15:14:09.072665 917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790435318278681-12427-0/minicluster-data/master-0/wal/instance:
uuid: "1a93ab6102f94c8db0faad5f623b8486"
format_stamp: "Formatted at 2026-09-26 15:14:09 on dist-test-slave-865v"
server_key: "14560114da1fcc39969bd792ea094cab"
server_key_iv: "bd753c0b6dcf368cc37c3bcdadbc6e88"
server_key_version: "encryptionkey@0"
I20260926 15:14:09.078943 917 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.003s
I20260926 15:14:09.082902 932 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:14:09.084529 917 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 15:14:09.084756 917 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790435318278681-12427-0/minicluster-data/master-0/data,/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790435318278681-12427-0/minicluster-data/master-0/wal
uuid: "1a93ab6102f94c8db0faad5f623b8486"
format_stamp: "Formatted at 2026-09-26 15:14:09 on dist-test-slave-865v"
server_key: "14560114da1fcc39969bd792ea094cab"
server_key_iv: "bd753c0b6dcf368cc37c3bcdadbc6e88"
server_key_version: "encryptionkey@0"
I20260926 15:14:09.084975 917 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790435318278681-12427-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790435318278681-12427-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790435318278681-12427-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:14:09.091530 917 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:14:09.096189 917 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:14:09.096491 917 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:14:09.114925 917 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37429
I20260926 15:14:09.114971 983 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37429 every 8 connection(s)
I20260926 15:14:09.116245 917 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
I20260926 15:14:09.119828 984 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:14:09.123112 12427 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8uoJGv/build/asan/bin/kudu as pid 917
I20260926 15:14:09.123286 12427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790435318278681-12427-0/minicluster-data/master-0/wal/instance
I20260926 15:14:09.123814 12427 external_mini_cluster.cc:1428] Setting key 3e7c2b3ef035e613bcb1fdb8c0236681
I20260926 15:14:09.131624 987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:14:09.125733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58415 (local address 127.0.0.1:37429)
0926 15:14:09.126483 (+ 750us) server_negotiation.cc:206] Beginning negotiation
0926 15:14:09.126493 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 15:14:09.126533 (+ 40us) server_negotiation.cc:407] Connection header received
0926 15:14:09.128244 (+ 1711us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:14:09.128266 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:14:09.128634 (+ 368us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:14:09.129044 (+ 410us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:14:09.130106 (+ 1062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:14:09.130115 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:14:09.130127 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:14:09.130139 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:14:09.130190 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:14:09.130211 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:14:09.130432 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0926 15:14:09.130572 (+ 140us) server_negotiation.cc:299] Negotiation successful
0926 15:14:09.130743 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":560,"thread_start_us":348,"threads_started":1}
I20260926 15:14:09.131985 984 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a93ab6102f94c8db0faad5f623b8486: Bootstrap starting.
I20260926 15:14:09.135767 984 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1a93ab6102f94c8db0faad5f623b8486: Neither blocks nor log segments found. Creating new log.
I20260926 15:14:09.135965 12427 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 15:14:09.137056 984 log.cc:824] T 00000000000000000000000000000000 P 1a93ab6102f94c8db0faad5f623b8486: Log is configured to *not* fsync() on all Append() calls
I20260926 15:14:09.140841 984 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a93ab6102f94c8db0faad5f623b8486: No bootstrap required, opened a new log
I20260926 15:14:09.144296 987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:14:09.141551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46147 (local address 127.0.0.1:37429)
0926 15:14:09.141745 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0926 15:14:09.141752 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:14:09.142134 (+ 382us) server_negotiation.cc:407] Connection header received
0926 15:14:09.142253 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:14:09.142258 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:14:09.142315 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:14:09.142434 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:14:09.143496 (+ 1062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:14:09.143502 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:14:09.143507 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:14:09.143526 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:14:09.143572 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:14:09.143592 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:14:09.143721 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0926 15:14:09.143896 (+ 175us) server_negotiation.cc:299] Negotiation successful
0926 15:14:09.143956 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 15:14:09.149426 984 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1a93ab6102f94c8db0faad5f623b8486 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a93ab6102f94c8db0faad5f623b8486" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37429 } }
I20260926 15:14:09.149812 984 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1a93ab6102f94c8db0faad5f623b8486 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:14:09.149919 984 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1a93ab6102f94c8db0faad5f623b8486 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a93ab6102f94c8db0faad5f623b8486, State: Initialized, Role: FOLLOWER
I20260926 15:14:09.150504 984 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1a93ab6102f94c8db0faad5f623b8486 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a93ab6102f94c8db0faad5f623b8486" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37429 } }
I20260926 15:14:09.150758 984 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1a93ab6102f94c8db0faad5f623b8486 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 15:14:09.150852 984 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1a93ab6102f94c8db0faad5f623b8486 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 15:14:09.151000 984 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1a93ab6102f94c8db0faad5f623b8486 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:14:09.153235 984 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1a93ab6102f94c8db0faad5f623b8486 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a93ab6102f94c8db0faad5f623b8486" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37429 } }
I20260926 15:14:09.153791 984 leader_election.cc:302] T 00000000000000000000000000000000 P 1a93ab6102f94c8db0faad5f623b8486 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1a93ab6102f94c8db0faad5f623b8486; no voters:
I20260926 15:14:09.154481 984 leader_election.cc:288] T 00000000000000000000000000000000 P 1a93ab6102f94c8db0faad5f623b8486 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 15:14:09.154616 994 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1a93ab6102f94c8db0faad5f623b8486 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:14:09.155294 994 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1a93ab6102f94c8db0faad5f623b8486 [term 1 LEADER]: Becoming Leader. State: Replica: 1a93ab6102f94c8db0faad5f623b8486, State: Running, Role: LEADER
I20260926 15:14:09.155845 994 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1a93ab6102f94c8db0faad5f623b8486 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a93ab6102f94c8db0faad5f623b8486" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37429 } }
I20260926 15:14:09.156554 984 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1a93ab6102f94c8db0faad5f623b8486 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:14:09.158779 995 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a93ab6102f94c8db0faad5f623b8486 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1a93ab6102f94c8db0faad5f623b8486" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a93ab6102f94c8db0faad5f623b8486" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37429 } } }
I20260926 15:14:09.158771 996 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a93ab6102f94c8db0faad5f623b8486 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1a93ab6102f94c8db0faad5f623b8486. Latest consensus state: current_term: 1 leader_uuid: "1a93ab6102f94c8db0faad5f623b8486" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a93ab6102f94c8db0faad5f623b8486" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37429 } } }
I20260926 15:14:09.159202 995 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a93ab6102f94c8db0faad5f623b8486 [sys.catalog]: This master's current role is: LEADER
I20260926 15:14:09.159268 996 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a93ab6102f94c8db0faad5f623b8486 [sys.catalog]: This master's current role is: LEADER
I20260926 15:14:09.162429 1001 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:14:09.167506 1001 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 15:14:09.177246 1001 catalog_manager.cc:1347] Generated new cluster ID: 34a41946114d4115866af43d42933fd7
I20260926 15:14:09.177419 1001 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 15:14:09.198932 1001 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 15:14:09.199437 1001 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:14:09.217716 1001 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1a93ab6102f94c8db0faad5f623b8486: Generated new TSK 0
I20260926 15:14:09.218689 1001 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 15:14:09.224153 12427 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 917
W20260926 15:14:09.225217 979 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:58415
W20260926 15:14:09.225972 12427 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 917: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 15:14:09.226135 12427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8uoJGv/build/asan/bin/kudu with pid 917
2026-09-26T15:14:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1929 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790435318278681-12427-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 15:14:09 dist-test-slave-865v krb5kdc[1015](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 15:14:09 dist-test-slave-865v krb5kdc[1015](info): set up 2 sockets
Sep 26 15:14:09 dist-test-slave-865v krb5kdc[1015](info): commencing operation
krb5kdc: starting...
W20260926 15:14:10.342060 12427 mini_kdc.cc:121] Time spent starting KDC: real 1.080s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790435318278681-12427-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790435318278681-12427-0/krb5kdc/test-user.keytab.
Sep 26 15:14:10 dist-test-slave-865v krb5kdc[1015](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435650, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T15:14:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T15:14:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790435318278681-12427-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790435318278681-12427-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 15:14:10.616626 12427 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8uoJGv/build/asan/bin/kudu
/tmp/dist-test-task8uoJGv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790435318278681-12427-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790435318278681-12427-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790435318278681-12427-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35661
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790435318278681-12427-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35661
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790435318278681-12427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790435318278681-12427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790435318278681-12427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:14:10.947805 1028 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:14:10.948112 1028 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:14:10.948205 1028 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:14:10.955595 1028 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:14:10.955718 1028 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:14:10.955771 1028 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:14:10.955824 1028 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:14:10.955860 1028 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:14:10.965919 1028 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790435318278681-12427-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790435318278681-12427-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35661
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790435318278681-12427-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35661
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790435318278681-12427-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:14:10.967161 1028 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:14:10.968369 1028 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:14:10.980566 1036 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:14:10.980592 1034 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:14:10.980721 1033 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:14:10.981856 1028 server_base.cc:1034] running on GCE node
I20260926 15:14:10.982656 1028 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:14:10.984370 1028 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:14:10.985620 1028 hybrid_clock.cc:648] HybridClock initialized: now 1790435650985574 us; error 60 us; skew 500 ppm
I20260926 15:14:10.985992 1028 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:14:10 dist-test-slave-865v krb5kdc[1015](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435650, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:14:11.000417 1028 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 15:14:11.002097 1028 webserver.cc:460] Webserver started at http://127.0.0.1:37115/ using document root <none> and password file <none>
I20260926 15:14:11.002804 1028 fs_manager.cc:359] Metadata directory not provided
I20260926 15:14:11.002955 1028 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:14:11.003266 1028 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:14:11.006398 1028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790435318278681-12427-0/minicluster-data/master-0/data/instance:
uuid: "a4d71e161b374e958eb484b2c7aeea59"
format_stamp: "Formatted at 2026-09-26 15:14:10 on dist-test-slave-865v"
server_key: "fe2ee0d01194254fb3f794f305ca0c77"
server_key_iv: "24fb3d2d08a8989da9cbe48477ab7e62"
server_key_version: "encryptionkey@0"
I20260926 15:14:11.007212 1028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790435318278681-12427-0/minicluster-data/master-0/wal/instance:
uuid: "a4d71e161b374e958eb484b2c7aeea59"
format_stamp: "Formatted at 2026-09-26 15:14:10 on dist-test-slave-865v"
server_key: "fe2ee0d01194254fb3f794f305ca0c77"
server_key_iv: "24fb3d2d08a8989da9cbe48477ab7e62"
server_key_version: "encryptionkey@0"
I20260926 15:14:11.012830 1028 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 15:14:11.016551 1043 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:14:11.017971 1028 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 15:14:11.018177 1028 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790435318278681-12427-0/minicluster-data/master-0/data,/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790435318278681-12427-0/minicluster-data/master-0/wal
uuid: "a4d71e161b374e958eb484b2c7aeea59"
format_stamp: "Formatted at 2026-09-26 15:14:10 on dist-test-slave-865v"
server_key: "fe2ee0d01194254fb3f794f305ca0c77"
server_key_iv: "24fb3d2d08a8989da9cbe48477ab7e62"
server_key_version: "encryptionkey@0"
I20260926 15:14:11.018385 1028 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790435318278681-12427-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790435318278681-12427-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790435318278681-12427-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:14:11.034142 1028 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:14:11.038446 1028 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:14:11.038782 1028 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:14:11.057358 1028 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35661
I20260926 15:14:11.057439 1094 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35661 every 8 connection(s)
I20260926 15:14:11.058700 1028 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
I20260926 15:14:11.062182 12427 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8uoJGv/build/asan/bin/kudu as pid 1028
I20260926 15:14:11.062367 12427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790435318278681-12427-0/minicluster-data/master-0/wal/instance
I20260926 15:14:11.062419 1095 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:14:11.062835 12427 external_mini_cluster.cc:1428] Setting key d404cafa3bbe0f6599ddbed92fe0265d
I20260926 15:14:11.076923 1095 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4d71e161b374e958eb484b2c7aeea59: Bootstrap starting.
Sep 26 15:14:11 dist-test-slave-865v krb5kdc[1015](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435650, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 15:14:11.080953 1095 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a4d71e161b374e958eb484b2c7aeea59: Neither blocks nor log segments found. Creating new log.
I20260926 15:14:11.082361 1095 log.cc:824] T 00000000000000000000000000000000 P a4d71e161b374e958eb484b2c7aeea59: Log is configured to *not* fsync() on all Append() calls
I20260926 15:14:11.085989 1095 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4d71e161b374e958eb484b2c7aeea59: No bootstrap required, opened a new log
I20260926 15:14:11.087994 1098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:14:11.064526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.240:35179 (local address 127.0.0.1:35661)
0926 15:14:11.065458 (+ 932us) server_negotiation.cc:206] Beginning negotiation
0926 15:14:11.065473 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 15:14:11.065520 (+ 47us) server_negotiation.cc:407] Connection header received
0926 15:14:11.067311 (+ 1791us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:14:11.067336 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:14:11.067829 (+ 493us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:14:11.068379 (+ 550us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:14:11.069764 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:14:11.071735 (+ 1971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:14:11.073040 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:14:11.074234 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:14:11.074428 (+ 194us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:14:11.078791 (+ 4363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:14:11.078804 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:14:11.078823 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:14:11.078887 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:14:11.082899 (+ 4012us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:14:11.083998 (+ 1099us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:14:11.084008 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:14:11.084016 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:14:11.084244 (+ 228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:14:11.084739 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:14:11.084746 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:14:11.084748 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:14:11.085457 (+ 709us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:14:11.085780 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0926 15:14:11.086281 (+ 501us) server_negotiation.cc:299] Negotiation successful
0926 15:14:11.086666 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":641,"thread_start_us":351,"threads_started":1}
I20260926 15:14:11.092077 12427 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 15:14:11.095014 1095 raft_consensus.cc:348] T 00000000000000000000000000000000 P a4d71e161b374e958eb484b2c7aeea59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4d71e161b374e958eb484b2c7aeea59" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35661 } }
I20260926 15:14:11.095367 1095 raft_consensus.cc:374] T 00000000000000000000000000000000 P a4d71e161b374e958eb484b2c7aeea59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:14:11.095484 1095 raft_consensus.cc:729] T 00000000000000000000000000000000 P a4d71e161b374e958eb484b2c7aeea59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4d71e161b374e958eb484b2c7aeea59, State: Initialized, Role: FOLLOWER
I20260926 15:14:11.096078 1095 consensus_queue.cc:260] T 00000000000000000000000000000000 P a4d71e161b374e958eb484b2c7aeea59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4d71e161b374e958eb484b2c7aeea59" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35661 } }
I20260926 15:14:11.096344 1095 raft_consensus.cc:388] T 00000000000000000000000000000000 P a4d71e161b374e958eb484b2c7aeea59 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 15:14:11.096510 1095 raft_consensus.cc:482] T 00000000000000000000000000000000 P a4d71e161b374e958eb484b2c7aeea59 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 15:14:11.096715 1095 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a4d71e161b374e958eb484b2c7aeea59 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:14:11.100062 1095 raft_consensus.cc:504] T 00000000000000000000000000000000 P a4d71e161b374e958eb484b2c7aeea59 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4d71e161b374e958eb484b2c7aeea59" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35661 } }
I20260926 15:14:11.100641 1095 leader_election.cc:302] T 00000000000000000000000000000000 P a4d71e161b374e958eb484b2c7aeea59 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a4d71e161b374e958eb484b2c7aeea59; no voters:
I20260926 15:14:11.101356 1095 leader_election.cc:288] T 00000000000000000000000000000000 P a4d71e161b374e958eb484b2c7aeea59 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 15:14:11.101476 1105 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a4d71e161b374e958eb484b2c7aeea59 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:14:11.102166 1105 raft_consensus.cc:686] T 00000000000000000000000000000000 P a4d71e161b374e958eb484b2c7aeea59 [term 1 LEADER]: Becoming Leader. State: Replica: a4d71e161b374e958eb484b2c7aeea59, State: Running, Role: LEADER
I20260926 15:14:11.102751 1105 consensus_queue.cc:237] T 00000000000000000000000000000000 P a4d71e161b374e958eb484b2c7aeea59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4d71e161b374e958eb484b2c7aeea59" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35661 } }
I20260926 15:14:11.104770 1105 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4d71e161b374e958eb484b2c7aeea59 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a4d71e161b374e958eb484b2c7aeea59. Latest consensus state: current_term: 1 leader_uuid: "a4d71e161b374e958eb484b2c7aeea59" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4d71e161b374e958eb484b2c7aeea59" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35661 } } }
I20260926 15:14:11.105105 1105 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4d71e161b374e958eb484b2c7aeea59 [sys.catalog]: This master's current role is: LEADER
I20260926 15:14:11.105223 1107 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4d71e161b374e958eb484b2c7aeea59 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a4d71e161b374e958eb484b2c7aeea59" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4d71e161b374e958eb484b2c7aeea59" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35661 } } }
I20260926 15:14:11.105583 1107 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4d71e161b374e958eb484b2c7aeea59 [sys.catalog]: This master's current role is: LEADER
I20260926 15:14:11.109835 1095 sys_catalog.cc:564] T 00000000000000000000000000000000 P a4d71e161b374e958eb484b2c7aeea59 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:14:11.113293 1109 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:14:11.115883 1098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:14:11.096595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.240:55295 (local address 127.0.0.1:35661)
0926 15:14:11.096766 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0926 15:14:11.096770 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:14:11.097431 (+ 661us) server_negotiation.cc:407] Connection header received
0926 15:14:11.097614 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:14:11.097618 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:14:11.097685 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:14:11.097765 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:14:11.099563 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:14:11.100382 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:14:11.102372 (+ 1990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:14:11.103313 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:14:11.103440 (+ 127us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:14:11.107599 (+ 4159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:14:11.107610 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:14:11.107617 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:14:11.107665 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:14:11.111645 (+ 3980us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:14:11.112872 (+ 1227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:14:11.112881 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:14:11.112887 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:14:11.113060 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:14:11.113769 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:14:11.113774 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:14:11.113778 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:14:11.114222 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:14:11.114508 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0926 15:14:11.115083 (+ 575us) server_negotiation.cc:299] Negotiation successful
0926 15:14:11.115336 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 15:14:11.119189 1109 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 15:14:11.129758 1109 catalog_manager.cc:1347] Generated new cluster ID: 579d1c83772a4eeda53a7407177ff1b2
I20260926 15:14:11.129895 1109 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 15:14:11.145823 1109 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 15:14:11.147644 1109 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:14:11.158843 1109 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a4d71e161b374e958eb484b2c7aeea59: Generated new TSK 0
I20260926 15:14:11.159886 1109 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 15:14:11.174551 12427 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 1028
I20260926 15:14:11.178727 1090 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:14:11.274803 12427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8uoJGv/build/asan/bin/kudu with pid 1028
W20260926 15:14:11.292922 1013 connection.cc:535] client connection to 127.0.0.1:35661 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35661): Connection reset by peer (error 104)
2026-09-26T15:14:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2049 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790435318278681-12427-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 15:14:11 dist-test-slave-865v krb5kdc[1127](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 15:14:11 dist-test-slave-865v krb5kdc[1127](info): set up 2 sockets
Sep 26 15:14:11 dist-test-slave-865v krb5kdc[1127](info): commencing operation
krb5kdc: starting...
W20260926 15:14:12.394953 12427 mini_kdc.cc:121] Time spent starting KDC: real 1.083s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790435318278681-12427-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790435318278681-12427-0/krb5kdc/test-user.keytab.
Sep 26 15:14:12 dist-test-slave-865v krb5kdc[1127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435652, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T15:14:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T15:14:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790435318278681-12427-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790435318278681-12427-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 15:14:12.656364 12427 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8uoJGv/build/asan/bin/kudu
/tmp/dist-test-task8uoJGv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790435318278681-12427-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790435318278681-12427-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790435318278681-12427-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41969
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790435318278681-12427-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41969
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790435318278681-12427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790435318278681-12427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790435318278681-12427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:14:12.985601 1140 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:14:12.985901 1140 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:14:12.985993 1140 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:14:12.993368 1140 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:14:12.993484 1140 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:14:12.993530 1140 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:14:12.993598 1140 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:14:12.993642 1140 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:14:13.003595 1140 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790435318278681-12427-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790435318278681-12427-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41969
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790435318278681-12427-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41969
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790435318278681-12427-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:14:13.004814 1140 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:14:13.006099 1140 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:14:13.017668 1148 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:14:13.017745 1145 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:14:13.017716 1146 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:14:13.018554 1140 server_base.cc:1034] running on GCE node
I20260926 15:14:13.019232 1140 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:14:13.020465 1140 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:14:13.021656 1140 hybrid_clock.cc:648] HybridClock initialized: now 1790435653021597 us; error 50 us; skew 500 ppm
I20260926 15:14:13.021998 1140 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:14:13 dist-test-slave-865v krb5kdc[1127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435653, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:14:13.025202 1140 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 15:14:13.026597 1140 webserver.cc:460] Webserver started at http://127.0.0.1:41805/ using document root <none> and password file <none>
I20260926 15:14:13.027220 1140 fs_manager.cc:359] Metadata directory not provided
I20260926 15:14:13.027345 1140 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:14:13.027640 1140 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:14:13.030602 1140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790435318278681-12427-0/minicluster-data/master-0/data/instance:
uuid: "c9d0423b5aee48a9a119f38823317197"
format_stamp: "Formatted at 2026-09-26 15:14:13 on dist-test-slave-865v"
server_key: "f4eb49e6428593938da98b7930821d8f"
server_key_iv: "2e0e1f819bd145fce6e374171e230105"
server_key_version: "encryptionkey@0"
I20260926 15:14:13.031368 1140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790435318278681-12427-0/minicluster-data/master-0/wal/instance:
uuid: "c9d0423b5aee48a9a119f38823317197"
format_stamp: "Formatted at 2026-09-26 15:14:13 on dist-test-slave-865v"
server_key: "f4eb49e6428593938da98b7930821d8f"
server_key_iv: "2e0e1f819bd145fce6e374171e230105"
server_key_version: "encryptionkey@0"
I20260926 15:14:13.036340 1140 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260926 15:14:13.039656 1155 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:14:13.040920 1140 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 15:14:13.041146 1140 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790435318278681-12427-0/minicluster-data/master-0/data,/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790435318278681-12427-0/minicluster-data/master-0/wal
uuid: "c9d0423b5aee48a9a119f38823317197"
format_stamp: "Formatted at 2026-09-26 15:14:13 on dist-test-slave-865v"
server_key: "f4eb49e6428593938da98b7930821d8f"
server_key_iv: "2e0e1f819bd145fce6e374171e230105"
server_key_version: "encryptionkey@0"
I20260926 15:14:13.041347 1140 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790435318278681-12427-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790435318278681-12427-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790435318278681-12427-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:14:13.057365 1140 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:14:13.061530 1140 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:14:13.061748 1140 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:14:13.080331 1140 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41969
I20260926 15:14:13.080377 1206 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41969 every 8 connection(s)
I20260926 15:14:13.081759 1140 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
I20260926 15:14:13.085568 1207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:14:13.090312 12427 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8uoJGv/build/asan/bin/kudu as pid 1140
I20260926 15:14:13.090497 12427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790435318278681-12427-0/minicluster-data/master-0/wal/instance
I20260926 15:14:13.091049 12427 external_mini_cluster.cc:1428] Setting key dec163cc68afb9b9a783a1531aa837a5
I20260926 15:14:13.097646 1207 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9d0423b5aee48a9a119f38823317197: Bootstrap starting.
I20260926 15:14:13.101246 1207 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c9d0423b5aee48a9a119f38823317197: Neither blocks nor log segments found. Creating new log.
I20260926 15:14:13.102478 1207 log.cc:824] T 00000000000000000000000000000000 P c9d0423b5aee48a9a119f38823317197: Log is configured to *not* fsync() on all Append() calls
I20260926 15:14:13.105580 1210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:14:13.092880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.240:49161 (local address 127.0.0.1:41969)
0926 15:14:13.093720 (+ 840us) server_negotiation.cc:206] Beginning negotiation
0926 15:14:13.093735 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 15:14:13.093775 (+ 40us) server_negotiation.cc:407] Connection header received
0926 15:14:13.096092 (+ 2317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:14:13.096116 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:14:13.096529 (+ 413us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:14:13.096853 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:14:13.097481 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:14:13.098888 (+ 1407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:14:13.100769 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:14:13.101587 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:14:13.101729 (+ 142us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:14:13.103074 (+ 1345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:14:13.103087 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:14:13.103110 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:14:13.103916 (+ 806us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:14:13.104002 (+ 86us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:14:13.104026 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:14:13.104140 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0926 15:14:13.104452 (+ 312us) server_negotiation.cc:299] Negotiation successful
0926 15:14:13.104618 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":620,"thread_start_us":428,"threads_started":1}
I20260926 15:14:13.105613 1207 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9d0423b5aee48a9a119f38823317197: No bootstrap required, opened a new log
I20260926 15:14:13.109424 12427 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 15:14:13.115824 1207 raft_consensus.cc:348] T 00000000000000000000000000000000 P c9d0423b5aee48a9a119f38823317197 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9d0423b5aee48a9a119f38823317197" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41969 } }
I20260926 15:14:13.116212 1207 raft_consensus.cc:374] T 00000000000000000000000000000000 P c9d0423b5aee48a9a119f38823317197 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:14:13.116344 1207 raft_consensus.cc:729] T 00000000000000000000000000000000 P c9d0423b5aee48a9a119f38823317197 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9d0423b5aee48a9a119f38823317197, State: Initialized, Role: FOLLOWER
I20260926 15:14:13.116997 1207 consensus_queue.cc:260] T 00000000000000000000000000000000 P c9d0423b5aee48a9a119f38823317197 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9d0423b5aee48a9a119f38823317197" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41969 } }
I20260926 15:14:13.117293 1207 raft_consensus.cc:388] T 00000000000000000000000000000000 P c9d0423b5aee48a9a119f38823317197 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 15:14:13.117441 1207 raft_consensus.cc:482] T 00000000000000000000000000000000 P c9d0423b5aee48a9a119f38823317197 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 15:14:13.117645 1207 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c9d0423b5aee48a9a119f38823317197 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:14:13.120309 1207 raft_consensus.cc:504] T 00000000000000000000000000000000 P c9d0423b5aee48a9a119f38823317197 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9d0423b5aee48a9a119f38823317197" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41969 } }
I20260926 15:14:13.120802 1207 leader_election.cc:302] T 00000000000000000000000000000000 P c9d0423b5aee48a9a119f38823317197 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c9d0423b5aee48a9a119f38823317197; no voters:
I20260926 15:14:13.121595 1207 leader_election.cc:288] T 00000000000000000000000000000000 P c9d0423b5aee48a9a119f38823317197 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 15:14:13.121668 1210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:14:13.114698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.240:44911 (local address 127.0.0.1:41969)
0926 15:14:13.114892 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0926 15:14:13.114898 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:14:13.115345 (+ 447us) server_negotiation.cc:407] Connection header received
0926 15:14:13.115486 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:14:13.115491 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:14:13.115573 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:14:13.115688 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:14:13.116236 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:14:13.117371 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:14:13.118741 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:14:13.119448 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:14:13.119577 (+ 129us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:14:13.120654 (+ 1077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:14:13.120660 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:14:13.120667 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:14:13.120691 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:14:13.120747 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:14:13.120763 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:14:13.120890 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0926 15:14:13.121158 (+ 268us) server_negotiation.cc:299] Negotiation successful
0926 15:14:13.121217 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 15:14:13.121719 1217 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c9d0423b5aee48a9a119f38823317197 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:14:13.123652 1207 sys_catalog.cc:564] T 00000000000000000000000000000000 P c9d0423b5aee48a9a119f38823317197 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:14:13.128504 1217 raft_consensus.cc:686] T 00000000000000000000000000000000 P c9d0423b5aee48a9a119f38823317197 [term 1 LEADER]: Becoming Leader. State: Replica: c9d0423b5aee48a9a119f38823317197, State: Running, Role: LEADER
I20260926 15:14:13.129012 1217 consensus_queue.cc:237] T 00000000000000000000000000000000 P c9d0423b5aee48a9a119f38823317197 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9d0423b5aee48a9a119f38823317197" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41969 } }
I20260926 15:14:13.131933 1218 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9d0423b5aee48a9a119f38823317197 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c9d0423b5aee48a9a119f38823317197" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9d0423b5aee48a9a119f38823317197" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41969 } } }
I20260926 15:14:13.132331 1218 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9d0423b5aee48a9a119f38823317197 [sys.catalog]: This master's current role is: LEADER
I20260926 15:14:13.132392 1219 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9d0423b5aee48a9a119f38823317197 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c9d0423b5aee48a9a119f38823317197. Latest consensus state: current_term: 1 leader_uuid: "c9d0423b5aee48a9a119f38823317197" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9d0423b5aee48a9a119f38823317197" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41969 } } }
I20260926 15:14:13.132755 1219 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9d0423b5aee48a9a119f38823317197 [sys.catalog]: This master's current role is: LEADER
W20260926 15:14:13.137652 1231 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c9d0423b5aee48a9a119f38823317197: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 15:14:13.137763 1231 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 15:14:13.137902 1232 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:14:13.139493 1232 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 15:14:13.147990 1232 catalog_manager.cc:1347] Generated new cluster ID: 47050f6d7701440d9ab7dbea5824a3b6
I20260926 15:14:13.148142 1232 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 15:14:13.170696 1232 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 15:14:13.172618 1232 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:14:13.179785 1232 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c9d0423b5aee48a9a119f38823317197: Generated new TSK 0
I20260926 15:14:13.180871 1232 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 15:14:13.190549 12427 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 1140
W20260926 15:14:13.192126 1202 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.240:49161
W20260926 15:14:13.193045 12427 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 1140: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 15:14:13.193246 12427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8uoJGv/build/asan/bin/kudu with pid 1140
W20260926 15:14:13.211283 1125 connection.cc:535] client connection to 127.0.0.1:41969 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41969): Connection reset by peer (error 104)
2026-09-26T15:14:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1918 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790435318278681-12427-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 15:14:13 dist-test-slave-865v krb5kdc[1238](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 15:14:13 dist-test-slave-865v krb5kdc[1238](info): set up 2 sockets
Sep 26 15:14:13 dist-test-slave-865v krb5kdc[1238](info): commencing operation
krb5kdc: starting...
W20260926 15:14:14.309654 12427 mini_kdc.cc:121] Time spent starting KDC: real 1.079s user 0.004s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790435318278681-12427-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790435318278681-12427-0/krb5kdc/test-user.keytab.
Sep 26 15:14:14 dist-test-slave-865v krb5kdc[1238](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435654, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T15:14:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T15:14:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790435318278681-12427-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790435318278681-12427-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 15:14:14.574954 12427 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8uoJGv/build/asan/bin/kudu
/tmp/dist-test-task8uoJGv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790435318278681-12427-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790435318278681-12427-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790435318278681-12427-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35001
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790435318278681-12427-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35001
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790435318278681-12427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790435318278681-12427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790435318278681-12427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:14:14.903228 1251 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:14:14.903578 1251 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:14:14.903760 1251 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:14:14.911194 1251 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:14:14.911352 1251 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:14:14.911454 1251 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:14:14.911558 1251 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:14:14.911633 1251 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:14:14.921391 1251 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790435318278681-12427-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790435318278681-12427-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35001
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790435318278681-12427-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35001
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790435318278681-12427-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:14:14.922679 1251 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:14:14.923966 1251 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:14:14.939745 1257 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:14:14.939845 1259 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:14:14.940063 1256 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:14:14.940905 1251 server_base.cc:1034] running on GCE node
I20260926 15:14:14.941777 1251 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:14:14.943282 1251 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:14:14.944545 1251 hybrid_clock.cc:648] HybridClock initialized: now 1790435654944493 us; error 41 us; skew 500 ppm
I20260926 15:14:14.944986 1251 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:14:14 dist-test-slave-865v krb5kdc[1238](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435654, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:14:14.948642 1251 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 15:14:14.950387 1251 webserver.cc:460] Webserver started at http://127.0.0.1:39867/ using document root <none> and password file <none>
I20260926 15:14:14.951102 1251 fs_manager.cc:359] Metadata directory not provided
I20260926 15:14:14.951256 1251 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:14:14.951562 1251 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:14:14.954552 1251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790435318278681-12427-0/minicluster-data/master-0/data/instance:
uuid: "b09237f8574d45919201440fda1dad28"
format_stamp: "Formatted at 2026-09-26 15:14:14 on dist-test-slave-865v"
server_key: "9159fb437c6107374886976d7ffcd9a1"
server_key_iv: "5f6e889d93aea7242cf0b3adac2cae5b"
server_key_version: "encryptionkey@0"
I20260926 15:14:14.955331 1251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790435318278681-12427-0/minicluster-data/master-0/wal/instance:
uuid: "b09237f8574d45919201440fda1dad28"
format_stamp: "Formatted at 2026-09-26 15:14:14 on dist-test-slave-865v"
server_key: "9159fb437c6107374886976d7ffcd9a1"
server_key_iv: "5f6e889d93aea7242cf0b3adac2cae5b"
server_key_version: "encryptionkey@0"
I20260926 15:14:14.960395 1251 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260926 15:14:14.963879 1266 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:14:14.965234 1251 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 15:14:14.965437 1251 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790435318278681-12427-0/minicluster-data/master-0/data,/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790435318278681-12427-0/minicluster-data/master-0/wal
uuid: "b09237f8574d45919201440fda1dad28"
format_stamp: "Formatted at 2026-09-26 15:14:14 on dist-test-slave-865v"
server_key: "9159fb437c6107374886976d7ffcd9a1"
server_key_iv: "5f6e889d93aea7242cf0b3adac2cae5b"
server_key_version: "encryptionkey@0"
I20260926 15:14:14.965642 1251 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790435318278681-12427-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790435318278681-12427-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790435318278681-12427-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:14:14.971613 1251 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:14:14.976317 1251 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:14:14.976680 1251 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:14:14.995838 1251 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35001
I20260926 15:14:14.995887 1317 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35001 every 8 connection(s)
I20260926 15:14:14.997253 1251 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
I20260926 15:14:15.000145 12427 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8uoJGv/build/asan/bin/kudu as pid 1251
I20260926 15:14:15.000312 12427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790435318278681-12427-0/minicluster-data/master-0/wal/instance
I20260926 15:14:15.000888 12427 external_mini_cluster.cc:1428] Setting key bb73d169564b2d1d62acbd4755d6f38b
I20260926 15:14:15.001283 1318 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:14:15.009684 1321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:14:15.002865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.240:41451 (local address 127.0.0.1:35001)
0926 15:14:15.003729 (+ 864us) server_negotiation.cc:206] Beginning negotiation
0926 15:14:15.003741 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 15:14:15.003783 (+ 42us) server_negotiation.cc:407] Connection header received
0926 15:14:15.005556 (+ 1773us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:14:15.005577 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:14:15.005930 (+ 353us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:14:15.006283 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:14:15.007818 (+ 1535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:14:15.007825 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:14:15.007837 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:14:15.007854 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:14:15.007941 (+ 87us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:14:15.007967 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:14:15.008183 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0926 15:14:15.008377 (+ 194us) server_negotiation.cc:299] Negotiation successful
0926 15:14:15.008592 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":623,"thread_start_us":408,"threads_started":1}
I20260926 15:14:15.013521 12427 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 15:14:15.018725 1318 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b09237f8574d45919201440fda1dad28: Bootstrap starting.
I20260926 15:14:15.021847 1321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:14:15.019182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.240:59201 (local address 127.0.0.1:35001)
0926 15:14:15.019375 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0926 15:14:15.019381 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:14:15.019755 (+ 374us) server_negotiation.cc:407] Connection header received
0926 15:14:15.019889 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:14:15.019894 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:14:15.019958 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:14:15.020075 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:14:15.021139 (+ 1064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:14:15.021144 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:14:15.021148 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:14:15.021166 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:14:15.021210 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:14:15.021228 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:14:15.021348 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0926 15:14:15.021466 (+ 118us) server_negotiation.cc:299] Negotiation successful
0926 15:14:15.021526 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 15:14:15.022691 1318 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b09237f8574d45919201440fda1dad28: Neither blocks nor log segments found. Creating new log.
I20260926 15:14:15.024071 1318 log.cc:824] T 00000000000000000000000000000000 P b09237f8574d45919201440fda1dad28: Log is configured to *not* fsync() on all Append() calls
I20260926 15:14:15.027405 1318 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b09237f8574d45919201440fda1dad28: No bootstrap required, opened a new log
I20260926 15:14:15.035640 1318 raft_consensus.cc:348] T 00000000000000000000000000000000 P b09237f8574d45919201440fda1dad28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b09237f8574d45919201440fda1dad28" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35001 } }
I20260926 15:14:15.036062 1318 raft_consensus.cc:374] T 00000000000000000000000000000000 P b09237f8574d45919201440fda1dad28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:14:15.036187 1318 raft_consensus.cc:729] T 00000000000000000000000000000000 P b09237f8574d45919201440fda1dad28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b09237f8574d45919201440fda1dad28, State: Initialized, Role: FOLLOWER
I20260926 15:14:15.036775 1318 consensus_queue.cc:260] T 00000000000000000000000000000000 P b09237f8574d45919201440fda1dad28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b09237f8574d45919201440fda1dad28" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35001 } }
I20260926 15:14:15.037045 1318 raft_consensus.cc:388] T 00000000000000000000000000000000 P b09237f8574d45919201440fda1dad28 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 15:14:15.037266 1318 raft_consensus.cc:482] T 00000000000000000000000000000000 P b09237f8574d45919201440fda1dad28 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 15:14:15.037448 1318 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b09237f8574d45919201440fda1dad28 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:14:15.039773 1318 raft_consensus.cc:504] T 00000000000000000000000000000000 P b09237f8574d45919201440fda1dad28 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b09237f8574d45919201440fda1dad28" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35001 } }
I20260926 15:14:15.040292 1318 leader_election.cc:302] T 00000000000000000000000000000000 P b09237f8574d45919201440fda1dad28 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b09237f8574d45919201440fda1dad28; no voters:
I20260926 15:14:15.040920 1318 leader_election.cc:288] T 00000000000000000000000000000000 P b09237f8574d45919201440fda1dad28 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 15:14:15.041103 1328 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b09237f8574d45919201440fda1dad28 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:14:15.041926 1328 raft_consensus.cc:686] T 00000000000000000000000000000000 P b09237f8574d45919201440fda1dad28 [term 1 LEADER]: Becoming Leader. State: Replica: b09237f8574d45919201440fda1dad28, State: Running, Role: LEADER
I20260926 15:14:15.042502 1328 consensus_queue.cc:237] T 00000000000000000000000000000000 P b09237f8574d45919201440fda1dad28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b09237f8574d45919201440fda1dad28" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35001 } }
I20260926 15:14:15.043213 1318 sys_catalog.cc:564] T 00000000000000000000000000000000 P b09237f8574d45919201440fda1dad28 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:14:15.045626 1329 sys_catalog.cc:455] T 00000000000000000000000000000000 P b09237f8574d45919201440fda1dad28 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b09237f8574d45919201440fda1dad28" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b09237f8574d45919201440fda1dad28" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35001 } } }
I20260926 15:14:15.045842 1330 sys_catalog.cc:455] T 00000000000000000000000000000000 P b09237f8574d45919201440fda1dad28 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b09237f8574d45919201440fda1dad28. Latest consensus state: current_term: 1 leader_uuid: "b09237f8574d45919201440fda1dad28" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b09237f8574d45919201440fda1dad28" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35001 } } }
I20260926 15:14:15.046044 1329 sys_catalog.cc:458] T 00000000000000000000000000000000 P b09237f8574d45919201440fda1dad28 [sys.catalog]: This master's current role is: LEADER
I20260926 15:14:15.046118 1330 sys_catalog.cc:458] T 00000000000000000000000000000000 P b09237f8574d45919201440fda1dad28 [sys.catalog]: This master's current role is: LEADER
I20260926 15:14:15.048262 1335 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:14:15.053553 1335 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 15:14:15.063423 1335 catalog_manager.cc:1347] Generated new cluster ID: 728e9f3c56264f6e838f999d7c6fed96
I20260926 15:14:15.063542 1335 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 15:14:15.091439 1335 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 15:14:15.091910 1335 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:14:15.103443 1335 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b09237f8574d45919201440fda1dad28: Generated new TSK 0
I20260926 15:14:15.104555 1335 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 15:14:15.118147 12427 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 1251
W20260926 15:14:15.119336 1313 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.240:41451
W20260926 15:14:15.120162 12427 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 1251: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 15:14:15.120437 12427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8uoJGv/build/asan/bin/kudu with pid 1251
2026-09-26T15:14:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1927 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790435318278681-12427-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 15:14:15 dist-test-slave-865v krb5kdc[1349](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 15:14:15 dist-test-slave-865v krb5kdc[1349](info): set up 2 sockets
Sep 26 15:14:15 dist-test-slave-865v krb5kdc[1349](info): commencing operation
krb5kdc: starting...
W20260926 15:14:16.237651 12427 mini_kdc.cc:121] Time spent starting KDC: real 1.080s user 0.009s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790435318278681-12427-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790435318278681-12427-0/krb5kdc/test-user.keytab.
Sep 26 15:14:16 dist-test-slave-865v krb5kdc[1349](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435656, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T15:14:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T15:14:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790435318278681-12427-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790435318278681-12427-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 15:14:16.493460 12427 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8uoJGv/build/asan/bin/kudu
/tmp/dist-test-task8uoJGv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790435318278681-12427-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790435318278681-12427-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790435318278681-12427-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38331
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790435318278681-12427-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38331
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790435318278681-12427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790435318278681-12427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790435318278681-12427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:14:16.822846 1362 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:14:16.823123 1362 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:14:16.823213 1362 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:14:16.830371 1362 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:14:16.830494 1362 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:14:16.830542 1362 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:14:16.830613 1362 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:14:16.830658 1362 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:14:16.840569 1362 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790435318278681-12427-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790435318278681-12427-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38331
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790435318278681-12427-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38331
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790435318278681-12427-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:14:16.841853 1362 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:14:16.843075 1362 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:14:16.854983 1368 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:14:16.855000 1370 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:14:16.855672 1362 server_base.cc:1034] running on GCE node
W20260926 15:14:16.855795 1367 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:14:16.856737 1362 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:14:16.858276 1362 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:14:16.859511 1362 hybrid_clock.cc:648] HybridClock initialized: now 1790435656859469 us; error 61 us; skew 500 ppm
I20260926 15:14:16.859869 1362 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:14:16 dist-test-slave-865v krb5kdc[1349](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435656, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:14:16.871313 1362 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 15:14:16.872987 1362 webserver.cc:460] Webserver started at http://127.0.0.1:44341/ using document root <none> and password file <none>
I20260926 15:14:16.873732 1362 fs_manager.cc:359] Metadata directory not provided
I20260926 15:14:16.873894 1362 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:14:16.874212 1362 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:14:16.877310 1362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790435318278681-12427-0/minicluster-data/master-0/data/instance:
uuid: "d510222405914d02aaaff0c71a889947"
format_stamp: "Formatted at 2026-09-26 15:14:16 on dist-test-slave-865v"
server_key: "4d48d316cf35d019d69e599b1731511e"
server_key_iv: "07643284bd419e9770fa1f7e57ed7d8e"
server_key_version: "encryptionkey@0"
I20260926 15:14:16.878113 1362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790435318278681-12427-0/minicluster-data/master-0/wal/instance:
uuid: "d510222405914d02aaaff0c71a889947"
format_stamp: "Formatted at 2026-09-26 15:14:16 on dist-test-slave-865v"
server_key: "4d48d316cf35d019d69e599b1731511e"
server_key_iv: "07643284bd419e9770fa1f7e57ed7d8e"
server_key_version: "encryptionkey@0"
I20260926 15:14:16.883246 1362 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 15:14:16.887215 1377 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:14:16.888701 1362 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 15:14:16.888888 1362 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790435318278681-12427-0/minicluster-data/master-0/data,/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790435318278681-12427-0/minicluster-data/master-0/wal
uuid: "d510222405914d02aaaff0c71a889947"
format_stamp: "Formatted at 2026-09-26 15:14:16 on dist-test-slave-865v"
server_key: "4d48d316cf35d019d69e599b1731511e"
server_key_iv: "07643284bd419e9770fa1f7e57ed7d8e"
server_key_version: "encryptionkey@0"
I20260926 15:14:16.889065 1362 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790435318278681-12427-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790435318278681-12427-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790435318278681-12427-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:14:16.906850 1362 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:14:16.911361 1362 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:14:16.911641 1362 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:14:16.930373 1362 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38331
I20260926 15:14:16.930409 1428 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38331 every 8 connection(s)
I20260926 15:14:16.931736 1362 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
I20260926 15:14:16.935345 1429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:14:16.938508 12427 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8uoJGv/build/asan/bin/kudu as pid 1362
I20260926 15:14:16.938679 12427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790435318278681-12427-0/minicluster-data/master-0/wal/instance
I20260926 15:14:16.939141 12427 external_mini_cluster.cc:1428] Setting key 6762f93ce51ffa33fcb473b13d1b7b34
I20260926 15:14:16.948791 1429 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d510222405914d02aaaff0c71a889947: Bootstrap starting.
I20260926 15:14:16.952553 1429 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d510222405914d02aaaff0c71a889947: Neither blocks nor log segments found. Creating new log.
I20260926 15:14:16.953680 1429 log.cc:824] T 00000000000000000000000000000000 P d510222405914d02aaaff0c71a889947: Log is configured to *not* fsync() on all Append() calls
Sep 26 15:14:16 dist-test-slave-865v krb5kdc[1349](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435656, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 15:14:16.956836 1429 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d510222405914d02aaaff0c71a889947: No bootstrap required, opened a new log
I20260926 15:14:16.963933 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:14:16.941267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.240:39771 (local address 127.0.0.1:38331)
0926 15:14:16.942027 (+ 760us) server_negotiation.cc:206] Beginning negotiation
0926 15:14:16.942043 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 15:14:16.942094 (+ 51us) server_negotiation.cc:407] Connection header received
0926 15:14:16.944448 (+ 2354us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:14:16.944478 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:14:16.945037 (+ 559us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:14:16.945529 (+ 492us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:14:16.946886 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:14:16.948814 (+ 1928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:14:16.950267 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:14:16.951334 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:14:16.951519 (+ 185us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:14:16.956143 (+ 4624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:14:16.956152 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:14:16.956166 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:14:16.956205 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:14:16.959446 (+ 3241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:14:16.960371 (+ 925us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:14:16.960378 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:14:16.960384 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:14:16.960569 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:14:16.961022 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:14:16.961025 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:14:16.961027 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:14:16.961635 (+ 608us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:14:16.961888 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0926 15:14:16.962492 (+ 604us) server_negotiation.cc:299] Negotiation successful
0926 15:14:16.962886 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":517,"thread_start_us":300,"threads_started":1}
I20260926 15:14:16.965832 1429 raft_consensus.cc:348] T 00000000000000000000000000000000 P d510222405914d02aaaff0c71a889947 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d510222405914d02aaaff0c71a889947" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38331 } }
I20260926 15:14:16.966303 1429 raft_consensus.cc:374] T 00000000000000000000000000000000 P d510222405914d02aaaff0c71a889947 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:14:16.966454 1429 raft_consensus.cc:729] T 00000000000000000000000000000000 P d510222405914d02aaaff0c71a889947 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d510222405914d02aaaff0c71a889947, State: Initialized, Role: FOLLOWER
I20260926 15:14:16.967146 1429 consensus_queue.cc:260] T 00000000000000000000000000000000 P d510222405914d02aaaff0c71a889947 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d510222405914d02aaaff0c71a889947" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38331 } }
I20260926 15:14:16.967415 1429 raft_consensus.cc:388] T 00000000000000000000000000000000 P d510222405914d02aaaff0c71a889947 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 15:14:16.967556 1429 raft_consensus.cc:482] T 00000000000000000000000000000000 P d510222405914d02aaaff0c71a889947 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 15:14:16.967742 1429 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d510222405914d02aaaff0c71a889947 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:14:16.968338 12427 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 15:14:16.971014 1429 raft_consensus.cc:504] T 00000000000000000000000000000000 P d510222405914d02aaaff0c71a889947 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d510222405914d02aaaff0c71a889947" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38331 } }
I20260926 15:14:16.971601 1429 leader_election.cc:302] T 00000000000000000000000000000000 P d510222405914d02aaaff0c71a889947 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d510222405914d02aaaff0c71a889947; no voters:
I20260926 15:14:16.972313 1429 leader_election.cc:288] T 00000000000000000000000000000000 P d510222405914d02aaaff0c71a889947 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 15:14:16.972447 1436 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d510222405914d02aaaff0c71a889947 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:14:16.973863 1436 raft_consensus.cc:686] T 00000000000000000000000000000000 P d510222405914d02aaaff0c71a889947 [term 1 LEADER]: Becoming Leader. State: Replica: d510222405914d02aaaff0c71a889947, State: Running, Role: LEADER
I20260926 15:14:16.974536 1436 consensus_queue.cc:237] T 00000000000000000000000000000000 P d510222405914d02aaaff0c71a889947 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d510222405914d02aaaff0c71a889947" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38331 } }
I20260926 15:14:16.974699 1429 sys_catalog.cc:564] T 00000000000000000000000000000000 P d510222405914d02aaaff0c71a889947 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:14:16.977430 1439 sys_catalog.cc:455] T 00000000000000000000000000000000 P d510222405914d02aaaff0c71a889947 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d510222405914d02aaaff0c71a889947" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d510222405914d02aaaff0c71a889947" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38331 } } }
I20260926 15:14:16.977838 1439 sys_catalog.cc:458] T 00000000000000000000000000000000 P d510222405914d02aaaff0c71a889947 [sys.catalog]: This master's current role is: LEADER
I20260926 15:14:16.978252 1440 sys_catalog.cc:455] T 00000000000000000000000000000000 P d510222405914d02aaaff0c71a889947 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d510222405914d02aaaff0c71a889947. Latest consensus state: current_term: 1 leader_uuid: "d510222405914d02aaaff0c71a889947" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d510222405914d02aaaff0c71a889947" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38331 } } }
I20260926 15:14:16.978529 1440 sys_catalog.cc:458] T 00000000000000000000000000000000 P d510222405914d02aaaff0c71a889947 [sys.catalog]: This master's current role is: LEADER
I20260926 15:14:16.986726 1445 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:14:16.992240 1445 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 15:14:16.994736 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:14:16.974306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.240:50419 (local address 127.0.0.1:38331)
0926 15:14:16.974483 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0926 15:14:16.974488 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:14:16.974713 (+ 225us) server_negotiation.cc:407] Connection header received
0926 15:14:16.974955 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:14:16.974961 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:14:16.975039 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:14:16.975150 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:14:16.977110 (+ 1960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:14:16.978374 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:14:16.980843 (+ 2469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:14:16.981812 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:14:16.981933 (+ 121us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:14:16.986043 (+ 4110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:14:16.986049 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:14:16.986053 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:14:16.986093 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:14:16.990296 (+ 4203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:14:16.991261 (+ 965us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:14:16.991267 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:14:16.991272 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:14:16.991415 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:14:16.991949 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:14:16.991955 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:14:16.991960 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:14:16.992444 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:14:16.993960 (+ 1516us) server_negotiation.cc:1035] Waiting for connection context
0926 15:14:16.994166 (+ 206us) server_negotiation.cc:299] Negotiation successful
0926 15:14:16.994374 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 15:14:17.002168 1445 catalog_manager.cc:1347] Generated new cluster ID: 463f97a75d8341639e9bae886908de87
I20260926 15:14:17.002322 1445 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 15:14:17.017676 1445 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 15:14:17.019100 1445 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:14:17.028038 1445 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d510222405914d02aaaff0c71a889947: Generated new TSK 0
I20260926 15:14:17.028919 1445 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 15:14:17.034322 12427 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 1362
I20260926 15:14:17.035425 1424 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:14:17.125032 12427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8uoJGv/build/asan/bin/kudu with pid 1362
W20260926 15:14:17.142133 1347 connection.cc:535] client connection to 127.0.0.1:38331 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38331): Connection reset by peer (error 104)
2026-09-26T15:14:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2003 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790435318278681-12427-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 15:14:17 dist-test-slave-865v krb5kdc[1461](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 15:14:17 dist-test-slave-865v krb5kdc[1461](info): set up 2 sockets
Sep 26 15:14:17 dist-test-slave-865v krb5kdc[1461](info): commencing operation
krb5kdc: starting...
W20260926 15:14:18.269727 12427 mini_kdc.cc:121] Time spent starting KDC: real 1.108s user 0.004s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790435318278681-12427-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790435318278681-12427-0/krb5kdc/test-user.keytab.
Sep 26 15:14:18 dist-test-slave-865v krb5kdc[1461](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435658, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T15:14:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T15:14:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790435318278681-12427-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790435318278681-12427-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 15:14:18.536645 12427 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8uoJGv/build/asan/bin/kudu
/tmp/dist-test-task8uoJGv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790435318278681-12427-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790435318278681-12427-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790435318278681-12427-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44291
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790435318278681-12427-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44291
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790435318278681-12427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790435318278681-12427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790435318278681-12427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:14:18.871991 1474 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:14:18.872359 1474 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:14:18.872556 1474 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:14:18.879988 1474 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:14:18.880117 1474 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:14:18.880220 1474 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:14:18.880311 1474 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:14:18.880378 1474 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:14:18.890439 1474 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790435318278681-12427-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790435318278681-12427-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44291
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790435318278681-12427-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44291
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790435318278681-12427-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:14:18.891776 1474 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:14:18.893054 1474 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:14:18.904774 1479 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:14:18.904840 1480 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:14:18.904863 1482 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:14:18.905893 1474 server_base.cc:1034] running on GCE node
I20260926 15:14:18.906738 1474 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:14:18.908057 1474 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:14:18.909262 1474 hybrid_clock.cc:648] HybridClock initialized: now 1790435658909231 us; error 38 us; skew 500 ppm
I20260926 15:14:18.909592 1474 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:14:18 dist-test-slave-865v krb5kdc[1461](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435658, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:14:18.912657 1474 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 15:14:18.914036 1474 webserver.cc:460] Webserver started at http://127.0.0.1:36373/ using document root <none> and password file <none>
I20260926 15:14:18.914625 1474 fs_manager.cc:359] Metadata directory not provided
I20260926 15:14:18.914729 1474 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:14:18.914965 1474 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:14:18.917805 1474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790435318278681-12427-0/minicluster-data/master-0/data/instance:
uuid: "6010ddf9140848a08102ad12826fa7f6"
format_stamp: "Formatted at 2026-09-26 15:14:18 on dist-test-slave-865v"
server_key: "4b3b6653331d8bdf701bccafba1366c3"
server_key_iv: "711b92f374d0fb6bf33476f39ca2cd7f"
server_key_version: "encryptionkey@0"
I20260926 15:14:18.918471 1474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790435318278681-12427-0/minicluster-data/master-0/wal/instance:
uuid: "6010ddf9140848a08102ad12826fa7f6"
format_stamp: "Formatted at 2026-09-26 15:14:18 on dist-test-slave-865v"
server_key: "4b3b6653331d8bdf701bccafba1366c3"
server_key_iv: "711b92f374d0fb6bf33476f39ca2cd7f"
server_key_version: "encryptionkey@0"
I20260926 15:14:18.923753 1474 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260926 15:14:18.927089 1489 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:14:18.928361 1474 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 15:14:18.928530 1474 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790435318278681-12427-0/minicluster-data/master-0/data,/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790435318278681-12427-0/minicluster-data/master-0/wal
uuid: "6010ddf9140848a08102ad12826fa7f6"
format_stamp: "Formatted at 2026-09-26 15:14:18 on dist-test-slave-865v"
server_key: "4b3b6653331d8bdf701bccafba1366c3"
server_key_iv: "711b92f374d0fb6bf33476f39ca2cd7f"
server_key_version: "encryptionkey@0"
I20260926 15:14:18.928709 1474 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790435318278681-12427-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790435318278681-12427-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790435318278681-12427-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:14:18.944962 1474 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:14:18.949518 1474 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:14:18.949810 1474 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:14:18.968658 1474 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44291
I20260926 15:14:18.968706 1540 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44291 every 8 connection(s)
I20260926 15:14:18.970012 1474 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
I20260926 15:14:18.972805 12427 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8uoJGv/build/asan/bin/kudu as pid 1474
I20260926 15:14:18.972999 12427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790435318278681-12427-0/minicluster-data/master-0/wal/instance
I20260926 15:14:18.973537 12427 external_mini_cluster.cc:1428] Setting key 61114c791937a1f55a31e68590394ce9
I20260926 15:14:18.974207 1541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:14:18.986591 1544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:14:18.975297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.240:33503 (local address 127.0.0.1:44291)
0926 15:14:18.976030 (+ 733us) server_negotiation.cc:206] Beginning negotiation
0926 15:14:18.976041 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 15:14:18.976080 (+ 39us) server_negotiation.cc:407] Connection header received
0926 15:14:18.977618 (+ 1538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:14:18.977637 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:14:18.978003 (+ 366us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:14:18.978316 (+ 313us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:14:18.978902 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:14:18.980219 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:14:18.982060 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:14:18.982692 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:14:18.982825 (+ 133us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:14:18.984155 (+ 1330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:14:18.984163 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:14:18.984187 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:14:18.984736 (+ 549us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:14:18.984819 (+ 83us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:14:18.984846 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:14:18.984972 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0926 15:14:18.985426 (+ 454us) server_negotiation.cc:299] Negotiation successful
0926 15:14:18.985631 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":546,"thread_start_us":350,"threads_started":1}
I20260926 15:14:18.987607 1541 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6010ddf9140848a08102ad12826fa7f6: Bootstrap starting.
I20260926 15:14:18.990545 12427 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 15:14:18.991227 1541 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6010ddf9140848a08102ad12826fa7f6: Neither blocks nor log segments found. Creating new log.
I20260926 15:14:18.992422 1541 log.cc:824] T 00000000000000000000000000000000 P 6010ddf9140848a08102ad12826fa7f6: Log is configured to *not* fsync() on all Append() calls
I20260926 15:14:18.996237 1541 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6010ddf9140848a08102ad12826fa7f6: No bootstrap required, opened a new log
I20260926 15:14:19.002583 1544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:14:18.995870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.240:36083 (local address 127.0.0.1:44291)
0926 15:14:18.996215 (+ 345us) server_negotiation.cc:206] Beginning negotiation
0926 15:14:18.996226 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 15:14:18.996287 (+ 61us) server_negotiation.cc:407] Connection header received
0926 15:14:18.996518 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:14:18.996524 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:14:18.996617 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:14:18.996740 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:14:18.997266 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:14:18.998397 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:14:18.999665 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:14:19.000357 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:14:19.000495 (+ 138us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:14:19.001679 (+ 1184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:14:19.001692 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:14:19.001697 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:14:19.001717 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:14:19.001782 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:14:19.001806 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:14:19.001921 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0926 15:14:19.002155 (+ 234us) server_negotiation.cc:299] Negotiation successful
0926 15:14:19.002208 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":208}
I20260926 15:14:19.005808 1541 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6010ddf9140848a08102ad12826fa7f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6010ddf9140848a08102ad12826fa7f6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44291 } }
I20260926 15:14:19.006150 1541 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6010ddf9140848a08102ad12826fa7f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:14:19.006228 1541 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6010ddf9140848a08102ad12826fa7f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6010ddf9140848a08102ad12826fa7f6, State: Initialized, Role: FOLLOWER
I20260926 15:14:19.006775 1541 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6010ddf9140848a08102ad12826fa7f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6010ddf9140848a08102ad12826fa7f6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44291 } }
I20260926 15:14:19.006976 1541 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6010ddf9140848a08102ad12826fa7f6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 15:14:19.007164 1541 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6010ddf9140848a08102ad12826fa7f6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 15:14:19.007336 1541 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6010ddf9140848a08102ad12826fa7f6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:14:19.009570 1541 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6010ddf9140848a08102ad12826fa7f6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6010ddf9140848a08102ad12826fa7f6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44291 } }
I20260926 15:14:19.010108 1541 leader_election.cc:302] T 00000000000000000000000000000000 P 6010ddf9140848a08102ad12826fa7f6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6010ddf9140848a08102ad12826fa7f6; no voters:
I20260926 15:14:19.010851 1541 leader_election.cc:288] T 00000000000000000000000000000000 P 6010ddf9140848a08102ad12826fa7f6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 15:14:19.010957 1551 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6010ddf9140848a08102ad12826fa7f6 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:14:19.011706 1551 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6010ddf9140848a08102ad12826fa7f6 [term 1 LEADER]: Becoming Leader. State: Replica: 6010ddf9140848a08102ad12826fa7f6, State: Running, Role: LEADER
I20260926 15:14:19.012310 1551 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6010ddf9140848a08102ad12826fa7f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6010ddf9140848a08102ad12826fa7f6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44291 } }
I20260926 15:14:19.013132 1541 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6010ddf9140848a08102ad12826fa7f6 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:14:19.015295 1553 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6010ddf9140848a08102ad12826fa7f6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6010ddf9140848a08102ad12826fa7f6. Latest consensus state: current_term: 1 leader_uuid: "6010ddf9140848a08102ad12826fa7f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6010ddf9140848a08102ad12826fa7f6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44291 } } }
I20260926 15:14:19.015395 1552 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6010ddf9140848a08102ad12826fa7f6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6010ddf9140848a08102ad12826fa7f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6010ddf9140848a08102ad12826fa7f6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44291 } } }
I20260926 15:14:19.015722 1553 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6010ddf9140848a08102ad12826fa7f6 [sys.catalog]: This master's current role is: LEADER
I20260926 15:14:19.015807 1552 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6010ddf9140848a08102ad12826fa7f6 [sys.catalog]: This master's current role is: LEADER
I20260926 15:14:19.018808 1556 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:14:19.024044 1556 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 15:14:19.032764 1556 catalog_manager.cc:1347] Generated new cluster ID: ece0530e36344aa89ad529c59ed3c93e
I20260926 15:14:19.032912 1556 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 15:14:19.042335 1556 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 15:14:19.043572 1556 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:14:19.052052 1556 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6010ddf9140848a08102ad12826fa7f6: Generated new TSK 0
I20260926 15:14:19.052829 1556 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 15:14:19.063478 12427 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 1474
W20260926 15:14:19.064584 1536 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.240:33503
W20260926 15:14:19.065343 12427 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 1474: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 15:14:19.065495 12427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8uoJGv/build/asan/bin/kudu with pid 1474
W20260926 15:14:19.082894 1459 connection.cc:535] client connection to 127.0.0.1:44291 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44291): Connection reset by peer (error 104)
2026-09-26T15:14:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1941 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790435318278681-12427-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 15:14:19 dist-test-slave-865v krb5kdc[1572](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 15:14:19 dist-test-slave-865v krb5kdc[1572](info): set up 2 sockets
Sep 26 15:14:19 dist-test-slave-865v krb5kdc[1572](info): commencing operation
krb5kdc: starting...
W20260926 15:14:20.219310 12427 mini_kdc.cc:121] Time spent starting KDC: real 1.117s user 0.005s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790435318278681-12427-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790435318278681-12427-0/krb5kdc/test-user.keytab.
Sep 26 15:14:20 dist-test-slave-865v krb5kdc[1572](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435660, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T15:14:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T15:14:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790435318278681-12427-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790435318278681-12427-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 15:14:20.484174 12427 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8uoJGv/build/asan/bin/kudu
/tmp/dist-test-task8uoJGv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790435318278681-12427-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790435318278681-12427-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790435318278681-12427-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34297
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790435318278681-12427-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34297
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790435318278681-12427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790435318278681-12427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790435318278681-12427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:14:20.813597 1585 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:14:20.813972 1585 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:14:20.814134 1585 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:14:20.821779 1585 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:14:20.821959 1585 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:14:20.822070 1585 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:14:20.822196 1585 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:14:20.822274 1585 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:14:20.832507 1585 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790435318278681-12427-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790435318278681-12427-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34297
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790435318278681-12427-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34297
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790435318278681-12427-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:14:20.833894 1585 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:14:20.835223 1585 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:14:20.847903 1591 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:14:20.847868 1593 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:14:20.847951 1590 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:14:20.848834 1585 server_base.cc:1034] running on GCE node
I20260926 15:14:20.849677 1585 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:14:20.851059 1585 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:14:20.852308 1585 hybrid_clock.cc:648] HybridClock initialized: now 1790435660852277 us; error 46 us; skew 500 ppm
I20260926 15:14:20.852661 1585 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:14:20 dist-test-slave-865v krb5kdc[1572](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435660, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:14:20.855929 1585 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 15:14:20.857445 1585 webserver.cc:460] Webserver started at http://127.0.0.1:35631/ using document root <none> and password file <none>
I20260926 15:14:20.858101 1585 fs_manager.cc:359] Metadata directory not provided
I20260926 15:14:20.858233 1585 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:14:20.858543 1585 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:14:20.861585 1585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790435318278681-12427-0/minicluster-data/master-0/data/instance:
uuid: "a76a0a73b94246a5aac3fe57f93024bf"
format_stamp: "Formatted at 2026-09-26 15:14:20 on dist-test-slave-865v"
server_key: "0c859f2b46c210609787d9b664acdf04"
server_key_iv: "27be5ffc5989179562c12715dd568fdc"
server_key_version: "encryptionkey@0"
I20260926 15:14:20.862344 1585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790435318278681-12427-0/minicluster-data/master-0/wal/instance:
uuid: "a76a0a73b94246a5aac3fe57f93024bf"
format_stamp: "Formatted at 2026-09-26 15:14:20 on dist-test-slave-865v"
server_key: "0c859f2b46c210609787d9b664acdf04"
server_key_iv: "27be5ffc5989179562c12715dd568fdc"
server_key_version: "encryptionkey@0"
I20260926 15:14:20.867632 1585 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 15:14:20.871793 1600 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:14:20.873365 1585 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 15:14:20.873560 1585 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790435318278681-12427-0/minicluster-data/master-0/data,/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790435318278681-12427-0/minicluster-data/master-0/wal
uuid: "a76a0a73b94246a5aac3fe57f93024bf"
format_stamp: "Formatted at 2026-09-26 15:14:20 on dist-test-slave-865v"
server_key: "0c859f2b46c210609787d9b664acdf04"
server_key_iv: "27be5ffc5989179562c12715dd568fdc"
server_key_version: "encryptionkey@0"
I20260926 15:14:20.873751 1585 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790435318278681-12427-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790435318278681-12427-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790435318278681-12427-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:14:20.879468 1585 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:14:20.883896 1585 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:14:20.884269 1585 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:14:20.904479 1585 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34297
I20260926 15:14:20.904520 1651 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34297 every 8 connection(s)
I20260926 15:14:20.905970 1585 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
I20260926 15:14:20.908793 12427 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8uoJGv/build/asan/bin/kudu as pid 1585
I20260926 15:14:20.908978 12427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790435318278681-12427-0/minicluster-data/master-0/wal/instance
I20260926 15:14:20.909499 12427 external_mini_cluster.cc:1428] Setting key 26afb5016ce83a4abdadf39c4e86f52e
I20260926 15:14:20.909829 1652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:14:20.918749 1655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:14:20.911273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.240:60527 (local address 127.0.0.1:34297)
0926 15:14:20.912026 (+ 753us) server_negotiation.cc:206] Beginning negotiation
0926 15:14:20.912040 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 15:14:20.912090 (+ 50us) server_negotiation.cc:407] Connection header received
0926 15:14:20.914436 (+ 2346us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:14:20.914462 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:14:20.914904 (+ 442us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:14:20.915358 (+ 454us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:14:20.916708 (+ 1350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:14:20.916717 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:14:20.916733 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:14:20.916757 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:14:20.916825 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:14:20.916856 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:14:20.917113 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0926 15:14:20.917247 (+ 134us) server_negotiation.cc:299] Negotiation successful
0926 15:14:20.917460 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":516,"thread_start_us":310,"threads_started":1}
I20260926 15:14:20.922493 12427 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 15:14:20.924440 1652 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a76a0a73b94246a5aac3fe57f93024bf: Bootstrap starting.
I20260926 15:14:20.927790 1652 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a76a0a73b94246a5aac3fe57f93024bf: Neither blocks nor log segments found. Creating new log.
I20260926 15:14:20.928970 1652 log.cc:824] T 00000000000000000000000000000000 P a76a0a73b94246a5aac3fe57f93024bf: Log is configured to *not* fsync() on all Append() calls
I20260926 15:14:20.929041 1655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:14:20.926798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.240:48729 (local address 127.0.0.1:34297)
0926 15:14:20.926979 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0926 15:14:20.926985 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:14:20.927238 (+ 253us) server_negotiation.cc:407] Connection header received
0926 15:14:20.927357 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:14:20.927362 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:14:20.927424 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:14:20.927521 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:14:20.928424 (+ 903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:14:20.928429 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:14:20.928434 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:14:20.928451 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:14:20.928494 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:14:20.928514 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:14:20.928652 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0926 15:14:20.928699 (+ 47us) server_negotiation.cc:299] Negotiation successful
0926 15:14:20.928762 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 15:14:20.931787 1652 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a76a0a73b94246a5aac3fe57f93024bf: No bootstrap required, opened a new log
I20260926 15:14:20.938715 1652 raft_consensus.cc:348] T 00000000000000000000000000000000 P a76a0a73b94246a5aac3fe57f93024bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a76a0a73b94246a5aac3fe57f93024bf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34297 } }
I20260926 15:14:20.939096 1652 raft_consensus.cc:374] T 00000000000000000000000000000000 P a76a0a73b94246a5aac3fe57f93024bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:14:20.939208 1652 raft_consensus.cc:729] T 00000000000000000000000000000000 P a76a0a73b94246a5aac3fe57f93024bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a76a0a73b94246a5aac3fe57f93024bf, State: Initialized, Role: FOLLOWER
I20260926 15:14:20.939801 1652 consensus_queue.cc:260] T 00000000000000000000000000000000 P a76a0a73b94246a5aac3fe57f93024bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a76a0a73b94246a5aac3fe57f93024bf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34297 } }
I20260926 15:14:20.940057 1652 raft_consensus.cc:388] T 00000000000000000000000000000000 P a76a0a73b94246a5aac3fe57f93024bf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 15:14:20.940210 1652 raft_consensus.cc:482] T 00000000000000000000000000000000 P a76a0a73b94246a5aac3fe57f93024bf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 15:14:20.940374 1652 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a76a0a73b94246a5aac3fe57f93024bf [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:14:20.942579 1652 raft_consensus.cc:504] T 00000000000000000000000000000000 P a76a0a73b94246a5aac3fe57f93024bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a76a0a73b94246a5aac3fe57f93024bf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34297 } }
I20260926 15:14:20.943076 1652 leader_election.cc:302] T 00000000000000000000000000000000 P a76a0a73b94246a5aac3fe57f93024bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a76a0a73b94246a5aac3fe57f93024bf; no voters:
I20260926 15:14:20.943755 1652 leader_election.cc:288] T 00000000000000000000000000000000 P a76a0a73b94246a5aac3fe57f93024bf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 15:14:20.943866 1662 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a76a0a73b94246a5aac3fe57f93024bf [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:14:20.944460 1662 raft_consensus.cc:686] T 00000000000000000000000000000000 P a76a0a73b94246a5aac3fe57f93024bf [term 1 LEADER]: Becoming Leader. State: Replica: a76a0a73b94246a5aac3fe57f93024bf, State: Running, Role: LEADER
I20260926 15:14:20.945104 1662 consensus_queue.cc:237] T 00000000000000000000000000000000 P a76a0a73b94246a5aac3fe57f93024bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a76a0a73b94246a5aac3fe57f93024bf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34297 } }
I20260926 15:14:20.946030 1652 sys_catalog.cc:564] T 00000000000000000000000000000000 P a76a0a73b94246a5aac3fe57f93024bf [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:14:20.948021 1663 sys_catalog.cc:455] T 00000000000000000000000000000000 P a76a0a73b94246a5aac3fe57f93024bf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a76a0a73b94246a5aac3fe57f93024bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a76a0a73b94246a5aac3fe57f93024bf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34297 } } }
I20260926 15:14:20.948145 1664 sys_catalog.cc:455] T 00000000000000000000000000000000 P a76a0a73b94246a5aac3fe57f93024bf [sys.catalog]: SysCatalogTable state changed. Reason: New leader a76a0a73b94246a5aac3fe57f93024bf. Latest consensus state: current_term: 1 leader_uuid: "a76a0a73b94246a5aac3fe57f93024bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a76a0a73b94246a5aac3fe57f93024bf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34297 } } }
I20260926 15:14:20.948527 1663 sys_catalog.cc:458] T 00000000000000000000000000000000 P a76a0a73b94246a5aac3fe57f93024bf [sys.catalog]: This master's current role is: LEADER
I20260926 15:14:20.948523 1664 sys_catalog.cc:458] T 00000000000000000000000000000000 P a76a0a73b94246a5aac3fe57f93024bf [sys.catalog]: This master's current role is: LEADER
I20260926 15:14:20.950305 1668 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:14:20.955220 1668 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 15:14:20.965616 1668 catalog_manager.cc:1347] Generated new cluster ID: 615fd76d9d924af889174ca1291d5fa2
I20260926 15:14:20.965821 1668 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 15:14:20.993625 1668 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 15:14:20.994074 1668 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:14:21.003332 1668 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a76a0a73b94246a5aac3fe57f93024bf: Generated new TSK 0
I20260926 15:14:21.004220 1668 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 15:14:21.011262 12427 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 1585
W20260926 15:14:21.012375 1647 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.240:60527
W20260926 15:14:21.013130 12427 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 1585: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 15:14:21.013509 12427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8uoJGv/build/asan/bin/kudu with pid 1585
2026-09-26T15:14:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1947 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790435318278681-12427-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 15:14:21 dist-test-slave-865v krb5kdc[1683](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 15:14:21 dist-test-slave-865v krb5kdc[1683](info): set up 2 sockets
Sep 26 15:14:21 dist-test-slave-865v krb5kdc[1683](info): commencing operation
krb5kdc: starting...
W20260926 15:14:22.126004 12427 mini_kdc.cc:121] Time spent starting KDC: real 1.076s user 0.005s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790435318278681-12427-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790435318278681-12427-0/krb5kdc/test-user.keytab.
Sep 26 15:14:22 dist-test-slave-865v krb5kdc[1683](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435662, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T15:14:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T15:14:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.34.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.34.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.34.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.254.keytab.
Entry for principal kudu/127.12.34.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.254.keytab.
I20260926 15:14:22.389164 12427 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8uoJGv/build/asan/bin/kudu
/tmp/dist-test-task8uoJGv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790435318278681-12427-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790435318278681-12427-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790435318278681-12427-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.34.254:46445
--webserver_interface=127.12.34.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.254.keytab
--principal=kudu/127.12.34.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.34.212:45217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.34.254:46445
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790435318278681-12427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790435318278681-12427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790435318278681-12427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:14:22.733357 1696 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:14:22.733697 1696 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:14:22.733872 1696 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:14:22.741096 1696 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:14:22.741272 1696 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:14:22.741380 1696 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:14:22.741492 1696 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:14:22.741585 1696 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:14:22.751777 1696 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.34.212:45217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790435318278681-12427-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790435318278681-12427-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.34.254:46445
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.254.keytab
--principal=kudu/127.12.34.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.34.254:46445
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.34.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790435318278681-12427-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:14:22.753198 1696 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:14:22.754513 1696 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:14:22.766078 1701 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:14:22.766031 1702 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:14:22.765996 1704 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:14:22.767089 1696 server_base.cc:1034] running on GCE node
I20260926 15:14:22.767820 1696 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:14:22.769224 1696 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:14:22.770480 1696 hybrid_clock.cc:648] HybridClock initialized: now 1790435662770424 us; error 65 us; skew 500 ppm
I20260926 15:14:22.770838 1696 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:14:22 dist-test-slave-865v krb5kdc[1683](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435662, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:14:22.773938 1696 init.cc:377] Logged in from keytab as kudu/127.12.34.254@KRBTEST.COM (short username kudu)
I20260926 15:14:22.775486 1696 webserver.cc:460] Webserver started at http://127.12.34.254:40879/ using document root <none> and password file <none>
I20260926 15:14:22.776137 1696 fs_manager.cc:359] Metadata directory not provided
I20260926 15:14:22.776286 1696 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:14:22.776572 1696 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:14:22.779515 1696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790435318278681-12427-0/minicluster-data/master-0/data/instance:
uuid: "778d72c9d96e4cbfb339f74086983048"
format_stamp: "Formatted at 2026-09-26 15:14:22 on dist-test-slave-865v"
server_key: "3e6e8f669a174c122d596baba523e913"
server_key_iv: "cb75978458aeb49d95d6e5711ff2b6cb"
server_key_version: "encryptionkey@0"
I20260926 15:14:22.780333 1696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790435318278681-12427-0/minicluster-data/master-0/wal/instance:
uuid: "778d72c9d96e4cbfb339f74086983048"
format_stamp: "Formatted at 2026-09-26 15:14:22 on dist-test-slave-865v"
server_key: "3e6e8f669a174c122d596baba523e913"
server_key_iv: "cb75978458aeb49d95d6e5711ff2b6cb"
server_key_version: "encryptionkey@0"
I20260926 15:14:22.785581 1696 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260926 15:14:22.788982 1711 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:14:22.790292 1696 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 15:14:22.790479 1696 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790435318278681-12427-0/minicluster-data/master-0/data,/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790435318278681-12427-0/minicluster-data/master-0/wal
uuid: "778d72c9d96e4cbfb339f74086983048"
format_stamp: "Formatted at 2026-09-26 15:14:22 on dist-test-slave-865v"
server_key: "3e6e8f669a174c122d596baba523e913"
server_key_iv: "cb75978458aeb49d95d6e5711ff2b6cb"
server_key_version: "encryptionkey@0"
I20260926 15:14:22.790675 1696 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790435318278681-12427-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790435318278681-12427-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790435318278681-12427-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:14:22.806265 1696 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:14:22.810227 1696 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:14:22.810518 1696 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:14:22.829716 1696 rpc_server.cc:307] RPC server started. Bound to: 127.12.34.254:46445
I20260926 15:14:22.829757 1762 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.34.254:46445 every 8 connection(s)
I20260926 15:14:22.831033 1696 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
I20260926 15:14:22.834517 12427 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8uoJGv/build/asan/bin/kudu as pid 1696
I20260926 15:14:22.834666 12427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790435318278681-12427-0/minicluster-data/master-0/wal/instance
I20260926 15:14:22.834903 1763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:14:22.835186 12427 external_mini_cluster.cc:1428] Setting key 1444a54cb03d6638077341818f09c339
I20260926 15:14:22.847918 1763 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 778d72c9d96e4cbfb339f74086983048: Bootstrap starting.
Sep 26 15:14:22 dist-test-slave-865v krb5kdc[1683](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435662, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.34.254@KRBTEST.COM
I20260926 15:14:22.851333 1763 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 778d72c9d96e4cbfb339f74086983048: Neither blocks nor log segments found. Creating new log.
I20260926 15:14:22.852524 1763 log.cc:824] T 00000000000000000000000000000000 P 778d72c9d96e4cbfb339f74086983048: Log is configured to *not* fsync() on all Append() calls
I20260926 15:14:22.855345 1763 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 778d72c9d96e4cbfb339f74086983048: No bootstrap required, opened a new log
I20260926 15:14:22.858927 1766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:14:22.836947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36525 (local address 127.12.34.254:46445)
0926 15:14:22.838143 (+ 1196us) server_negotiation.cc:206] Beginning negotiation
0926 15:14:22.838165 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0926 15:14:22.838235 (+ 70us) server_negotiation.cc:407] Connection header received
0926 15:14:22.840225 (+ 1990us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:14:22.840254 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:14:22.840727 (+ 473us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:14:22.841199 (+ 472us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:14:22.842668 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:14:22.843940 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:14:22.845267 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:14:22.845991 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:14:22.846188 (+ 197us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:14:22.850731 (+ 4543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:14:22.850739 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:14:22.850753 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:14:22.850794 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:14:22.854286 (+ 3492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:14:22.855175 (+ 889us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:14:22.855182 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:14:22.855188 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:14:22.855378 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:14:22.855941 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:14:22.855945 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:14:22.855947 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:14:22.856478 (+ 531us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:14:22.856843 (+ 365us) server_negotiation.cc:1035] Waiting for connection context
0926 15:14:22.857396 (+ 553us) server_negotiation.cc:299] Negotiation successful
0926 15:14:22.857773 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":813,"thread_start_us":477,"threads_started":1}
I20260926 15:14:22.863198 12427 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 15:14:22.864183 1763 raft_consensus.cc:348] T 00000000000000000000000000000000 P 778d72c9d96e4cbfb339f74086983048 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "778d72c9d96e4cbfb339f74086983048" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 46445 } }
I20260926 15:14:22.864539 1763 raft_consensus.cc:374] T 00000000000000000000000000000000 P 778d72c9d96e4cbfb339f74086983048 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:14:22.864643 1763 raft_consensus.cc:729] T 00000000000000000000000000000000 P 778d72c9d96e4cbfb339f74086983048 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 778d72c9d96e4cbfb339f74086983048, State: Initialized, Role: FOLLOWER
I20260926 15:14:22.865234 1763 consensus_queue.cc:260] T 00000000000000000000000000000000 P 778d72c9d96e4cbfb339f74086983048 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "778d72c9d96e4cbfb339f74086983048" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 46445 } }
I20260926 15:14:22.865477 1763 raft_consensus.cc:388] T 00000000000000000000000000000000 P 778d72c9d96e4cbfb339f74086983048 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 15:14:22.865615 1763 raft_consensus.cc:482] T 00000000000000000000000000000000 P 778d72c9d96e4cbfb339f74086983048 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 15:14:22.865778 1763 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 778d72c9d96e4cbfb339f74086983048 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:14:22.868155 1763 raft_consensus.cc:504] T 00000000000000000000000000000000 P 778d72c9d96e4cbfb339f74086983048 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "778d72c9d96e4cbfb339f74086983048" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 46445 } }
I20260926 15:14:22.868672 1763 leader_election.cc:302] T 00000000000000000000000000000000 P 778d72c9d96e4cbfb339f74086983048 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 778d72c9d96e4cbfb339f74086983048; no voters:
I20260926 15:14:22.869310 1763 leader_election.cc:288] T 00000000000000000000000000000000 P 778d72c9d96e4cbfb339f74086983048 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 15:14:22.869424 1773 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 778d72c9d96e4cbfb339f74086983048 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:14:22.871965 1763 sys_catalog.cc:564] T 00000000000000000000000000000000 P 778d72c9d96e4cbfb339f74086983048 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:14:22.876114 1773 raft_consensus.cc:686] T 00000000000000000000000000000000 P 778d72c9d96e4cbfb339f74086983048 [term 1 LEADER]: Becoming Leader. State: Replica: 778d72c9d96e4cbfb339f74086983048, State: Running, Role: LEADER
I20260926 15:14:22.876772 1773 consensus_queue.cc:237] T 00000000000000000000000000000000 P 778d72c9d96e4cbfb339f74086983048 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "778d72c9d96e4cbfb339f74086983048" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 46445 } }
I20260926 15:14:22.880544 1774 sys_catalog.cc:455] T 00000000000000000000000000000000 P 778d72c9d96e4cbfb339f74086983048 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "778d72c9d96e4cbfb339f74086983048" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "778d72c9d96e4cbfb339f74086983048" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 46445 } } }
I20260926 15:14:22.880976 1774 sys_catalog.cc:458] T 00000000000000000000000000000000 P 778d72c9d96e4cbfb339f74086983048 [sys.catalog]: This master's current role is: LEADER
I20260926 15:14:22.881510 1775 sys_catalog.cc:455] T 00000000000000000000000000000000 P 778d72c9d96e4cbfb339f74086983048 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 778d72c9d96e4cbfb339f74086983048. Latest consensus state: current_term: 1 leader_uuid: "778d72c9d96e4cbfb339f74086983048" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "778d72c9d96e4cbfb339f74086983048" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 46445 } } }
I20260926 15:14:22.881804 1775 sys_catalog.cc:458] T 00000000000000000000000000000000 P 778d72c9d96e4cbfb339f74086983048 [sys.catalog]: This master's current role is: LEADER
I20260926 15:14:22.886849 1766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:14:22.868211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55141 (local address 127.12.34.254:46445)
0926 15:14:22.868468 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0926 15:14:22.868478 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 15:14:22.869031 (+ 553us) server_negotiation.cc:407] Connection header received
0926 15:14:22.869298 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:14:22.869302 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:14:22.869369 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:14:22.869503 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:14:22.871456 (+ 1953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:14:22.872495 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:14:22.874041 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:14:22.874525 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:14:22.874662 (+ 137us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:14:22.877442 (+ 2780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:14:22.877448 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:14:22.877451 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:14:22.877482 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:14:22.882397 (+ 4915us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:14:22.883664 (+ 1267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:14:22.883673 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:14:22.883679 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:14:22.883858 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:14:22.884472 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:14:22.884482 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:14:22.884485 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:14:22.885032 (+ 547us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:14:22.885516 (+ 484us) server_negotiation.cc:1035] Waiting for connection context
0926 15:14:22.886057 (+ 541us) server_negotiation.cc:299] Negotiation successful
0926 15:14:22.886287 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 15:14:22.893592 1782 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:14:22.899283 1782 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 15:14:22.908432 1782 catalog_manager.cc:1347] Generated new cluster ID: 034c2c0c543b469c9a41b61e9fa6d1a1
I20260926 15:14:22.908591 1782 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 15:14:22.935658 1782 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 15:14:22.937674 1782 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:14:22.945961 1782 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 778d72c9d96e4cbfb339f74086983048: Generated new TSK 0
I20260926 15:14:22.946903 1782 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 15:14:22.960171 12427 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 1696
I20260926 15:14:22.961934 1758 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:14:23.055680 12427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8uoJGv/build/asan/bin/kudu with pid 1696
W20260926 15:14:23.072975 1681 connection.cc:535] client connection to 127.12.34.254:46445 recv error: Network error: failed to read from TLS socket (remote: 127.12.34.254:46445): Connection reset by peer (error 104)
2026-09-26T15:14:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2041 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790435318278681-12427-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 15:14:23 dist-test-slave-865v krb5kdc[1795](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 15:14:23 dist-test-slave-865v krb5kdc[1795](info): set up 2 sockets
Sep 26 15:14:23 dist-test-slave-865v krb5kdc[1795](info): commencing operation
krb5kdc: starting...
W20260926 15:14:24.213585 12427 mini_kdc.cc:121] Time spent starting KDC: real 1.122s user 0.007s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790435318278681-12427-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790435318278681-12427-0/krb5kdc/test-user.keytab.
Sep 26 15:14:24 dist-test-slave-865v krb5kdc[1795](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435664, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T15:14:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T15:14:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.34.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.34.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.34.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.254.keytab.
Entry for principal kudu/127.12.34.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.254.keytab.
I20260926 15:14:24.474625 12427 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8uoJGv/build/asan/bin/kudu
/tmp/dist-test-task8uoJGv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790435318278681-12427-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790435318278681-12427-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790435318278681-12427-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.34.254:43755
--webserver_interface=127.12.34.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.254.keytab
--principal=kudu/127.12.34.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.34.212:37431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.34.254:43755
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790435318278681-12427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790435318278681-12427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790435318278681-12427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:14:24.805928 1808 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:14:24.806252 1808 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:14:24.806433 1808 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:14:24.813714 1808 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:14:24.813829 1808 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:14:24.813943 1808 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:14:24.814052 1808 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:14:24.814141 1808 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:14:24.823812 1808 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.34.212:37431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790435318278681-12427-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790435318278681-12427-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.34.254:43755
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.254.keytab
--principal=kudu/127.12.34.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.34.254:43755
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.34.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790435318278681-12427-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:14:24.825096 1808 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:14:24.826380 1808 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:14:24.838245 1813 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:14:24.838296 1814 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:14:24.838337 1816 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:14:24.839136 1808 server_base.cc:1034] running on GCE node
I20260926 15:14:24.839830 1808 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:14:24.841235 1808 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:14:24.842456 1808 hybrid_clock.cc:648] HybridClock initialized: now 1790435664842423 us; error 31 us; skew 500 ppm
I20260926 15:14:24.842787 1808 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:14:24 dist-test-slave-865v krb5kdc[1795](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435664, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:14:24.845995 1808 init.cc:377] Logged in from keytab as kudu/127.12.34.254@KRBTEST.COM (short username kudu)
I20260926 15:14:24.847815 1808 webserver.cc:460] Webserver started at http://127.12.34.254:35051/ using document root <none> and password file <none>
I20260926 15:14:24.848541 1808 fs_manager.cc:359] Metadata directory not provided
I20260926 15:14:24.848649 1808 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:14:24.848989 1808 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:14:24.852169 1808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790435318278681-12427-0/minicluster-data/master-0/data/instance:
uuid: "5223445c43e44ec5b64a8091c358e68c"
format_stamp: "Formatted at 2026-09-26 15:14:24 on dist-test-slave-865v"
server_key: "d55f67526f02025b3cbdd52cfc190bc2"
server_key_iv: "b16fb18a58188b5ac3f611c41c08af66"
server_key_version: "encryptionkey@0"
I20260926 15:14:24.852958 1808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790435318278681-12427-0/minicluster-data/master-0/wal/instance:
uuid: "5223445c43e44ec5b64a8091c358e68c"
format_stamp: "Formatted at 2026-09-26 15:14:24 on dist-test-slave-865v"
server_key: "d55f67526f02025b3cbdd52cfc190bc2"
server_key_iv: "b16fb18a58188b5ac3f611c41c08af66"
server_key_version: "encryptionkey@0"
I20260926 15:14:24.858197 1808 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260926 15:14:24.861730 1823 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:14:24.863004 1808 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 15:14:24.863185 1808 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790435318278681-12427-0/minicluster-data/master-0/data,/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790435318278681-12427-0/minicluster-data/master-0/wal
uuid: "5223445c43e44ec5b64a8091c358e68c"
format_stamp: "Formatted at 2026-09-26 15:14:24 on dist-test-slave-865v"
server_key: "d55f67526f02025b3cbdd52cfc190bc2"
server_key_iv: "b16fb18a58188b5ac3f611c41c08af66"
server_key_version: "encryptionkey@0"
I20260926 15:14:24.863371 1808 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790435318278681-12427-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790435318278681-12427-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790435318278681-12427-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:14:24.875118 1808 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:14:24.879217 1808 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:14:24.879499 1808 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:14:24.898329 1808 rpc_server.cc:307] RPC server started. Bound to: 127.12.34.254:43755
I20260926 15:14:24.898366 1874 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.34.254:43755 every 8 connection(s)
I20260926 15:14:24.899701 1808 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
I20260926 15:14:24.902886 12427 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8uoJGv/build/asan/bin/kudu as pid 1808
I20260926 15:14:24.903120 12427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790435318278681-12427-0/minicluster-data/master-0/wal/instance
I20260926 15:14:24.903584 12427 external_mini_cluster.cc:1428] Setting key ff754d78452828711697ff06d63321e8
I20260926 15:14:24.903602 1875 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:14:24.917176 1878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:14:24.905238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34887 (local address 127.12.34.254:43755)
0926 15:14:24.905985 (+ 747us) server_negotiation.cc:206] Beginning negotiation
0926 15:14:24.905996 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 15:14:24.906042 (+ 46us) server_negotiation.cc:407] Connection header received
0926 15:14:24.907948 (+ 1906us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:14:24.907967 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:14:24.908334 (+ 367us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:14:24.908632 (+ 298us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:14:24.909326 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:14:24.910671 (+ 1345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:14:24.912638 (+ 1967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:14:24.913338 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:14:24.913459 (+ 121us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:14:24.914913 (+ 1454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:14:24.914921 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:14:24.914933 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:14:24.915470 (+ 537us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:14:24.915552 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:14:24.915580 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:14:24.915697 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0926 15:14:24.916017 (+ 320us) server_negotiation.cc:299] Negotiation successful
0926 15:14:24.916198 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":553,"thread_start_us":338,"threads_started":1}
I20260926 15:14:24.918249 1875 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5223445c43e44ec5b64a8091c358e68c: Bootstrap starting.
I20260926 15:14:24.921351 12427 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 15:14:24.921819 1875 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5223445c43e44ec5b64a8091c358e68c: Neither blocks nor log segments found. Creating new log.
I20260926 15:14:24.922825 1875 log.cc:824] T 00000000000000000000000000000000 P 5223445c43e44ec5b64a8091c358e68c: Log is configured to *not* fsync() on all Append() calls
I20260926 15:14:24.926222 1875 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5223445c43e44ec5b64a8091c358e68c: No bootstrap required, opened a new log
I20260926 15:14:24.932144 1878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:14:24.926039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60953 (local address 127.12.34.254:43755)
0926 15:14:24.926210 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0926 15:14:24.926241 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0926 15:14:24.926721 (+ 480us) server_negotiation.cc:407] Connection header received
0926 15:14:24.926927 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:14:24.926932 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:14:24.926997 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:14:24.927100 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:14:24.927595 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:14:24.928348 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:14:24.929705 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:14:24.930135 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:14:24.930334 (+ 199us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:14:24.931422 (+ 1088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:14:24.931427 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:14:24.931430 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:14:24.931446 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:14:24.931496 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:14:24.931511 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:14:24.931636 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0926 15:14:24.931831 (+ 195us) server_negotiation.cc:299] Negotiation successful
0926 15:14:24.931883 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 15:14:24.934617 1875 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5223445c43e44ec5b64a8091c358e68c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5223445c43e44ec5b64a8091c358e68c" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 43755 } }
I20260926 15:14:24.935024 1875 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5223445c43e44ec5b64a8091c358e68c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:14:24.935132 1875 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5223445c43e44ec5b64a8091c358e68c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5223445c43e44ec5b64a8091c358e68c, State: Initialized, Role: FOLLOWER
I20260926 15:14:24.935784 1875 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5223445c43e44ec5b64a8091c358e68c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5223445c43e44ec5b64a8091c358e68c" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 43755 } }
I20260926 15:14:24.936053 1875 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5223445c43e44ec5b64a8091c358e68c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 15:14:24.936205 1875 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5223445c43e44ec5b64a8091c358e68c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 15:14:24.936430 1875 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5223445c43e44ec5b64a8091c358e68c [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:14:24.938913 1875 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5223445c43e44ec5b64a8091c358e68c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5223445c43e44ec5b64a8091c358e68c" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 43755 } }
I20260926 15:14:24.939361 1875 leader_election.cc:302] T 00000000000000000000000000000000 P 5223445c43e44ec5b64a8091c358e68c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5223445c43e44ec5b64a8091c358e68c; no voters:
I20260926 15:14:24.939934 1875 leader_election.cc:288] T 00000000000000000000000000000000 P 5223445c43e44ec5b64a8091c358e68c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 15:14:24.940073 1885 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5223445c43e44ec5b64a8091c358e68c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:14:24.940773 1885 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5223445c43e44ec5b64a8091c358e68c [term 1 LEADER]: Becoming Leader. State: Replica: 5223445c43e44ec5b64a8091c358e68c, State: Running, Role: LEADER
I20260926 15:14:24.941344 1885 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5223445c43e44ec5b64a8091c358e68c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5223445c43e44ec5b64a8091c358e68c" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 43755 } }
I20260926 15:14:24.942284 1875 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5223445c43e44ec5b64a8091c358e68c [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:14:24.944411 1887 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5223445c43e44ec5b64a8091c358e68c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5223445c43e44ec5b64a8091c358e68c. Latest consensus state: current_term: 1 leader_uuid: "5223445c43e44ec5b64a8091c358e68c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5223445c43e44ec5b64a8091c358e68c" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 43755 } } }
I20260926 15:14:24.944896 1887 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5223445c43e44ec5b64a8091c358e68c [sys.catalog]: This master's current role is: LEADER
I20260926 15:14:24.945760 1886 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5223445c43e44ec5b64a8091c358e68c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5223445c43e44ec5b64a8091c358e68c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5223445c43e44ec5b64a8091c358e68c" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 43755 } } }
I20260926 15:14:24.946138 1886 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5223445c43e44ec5b64a8091c358e68c [sys.catalog]: This master's current role is: LEADER
I20260926 15:14:24.947244 1893 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:14:24.951879 1893 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 15:14:24.960632 1893 catalog_manager.cc:1347] Generated new cluster ID: 17b2ee34aa51432ab8e68a41a989a355
I20260926 15:14:24.960741 1893 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 15:14:24.987419 1893 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 15:14:24.988726 1893 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:14:24.997202 1893 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5223445c43e44ec5b64a8091c358e68c: Generated new TSK 0
I20260926 15:14:24.998057 1893 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 15:14:25.009230 12427 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 1808
W20260926 15:14:25.010541 1870 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:34887
W20260926 15:14:25.011292 12427 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 1808: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 15:14:25.011461 12427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8uoJGv/build/asan/bin/kudu with pid 1808
W20260926 15:14:25.028653 1793 connection.cc:535] client connection to 127.12.34.254:43755 recv error: Network error: failed to read from TLS socket (remote: 127.12.34.254:43755): Connection reset by peer (error 104)
2026-09-26T15:14:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1955 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790435318278681-12427-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 15:14:25 dist-test-slave-865v krb5kdc[1906](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 15:14:25 dist-test-slave-865v krb5kdc[1906](info): set up 2 sockets
Sep 26 15:14:25 dist-test-slave-865v krb5kdc[1906](info): commencing operation
krb5kdc: starting...
W20260926 15:14:26.193656 12427 mini_kdc.cc:121] Time spent starting KDC: real 1.146s user 0.000s sys 0.033s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790435318278681-12427-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790435318278681-12427-0/krb5kdc/test-user.keytab.
Sep 26 15:14:26 dist-test-slave-865v krb5kdc[1906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435666, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T15:14:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T15:14:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.34.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.34.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.34.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.254.keytab.
Entry for principal kudu/127.12.34.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.254.keytab.
I20260926 15:14:26.455842 12427 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8uoJGv/build/asan/bin/kudu
/tmp/dist-test-task8uoJGv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790435318278681-12427-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790435318278681-12427-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790435318278681-12427-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.34.254:34943
--webserver_interface=127.12.34.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.254.keytab
--principal=kudu/127.12.34.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.34.212:38411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.34.254:34943
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790435318278681-12427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790435318278681-12427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790435318278681-12427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:14:26.789705 1919 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:14:26.789992 1919 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:14:26.790091 1919 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:14:26.797380 1919 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:14:26.797494 1919 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:14:26.797540 1919 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:14:26.797600 1919 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:14:26.797643 1919 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:14:26.807610 1919 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.34.212:38411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790435318278681-12427-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790435318278681-12427-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.34.254:34943
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.254.keytab
--principal=kudu/127.12.34.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.34.254:34943
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.34.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790435318278681-12427-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:14:26.808836 1919 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:14:26.810104 1919 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:14:26.821988 1924 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:14:26.821941 1925 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:14:26.821889 1927 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:14:26.822818 1919 server_base.cc:1034] running on GCE node
I20260926 15:14:26.823504 1919 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:14:26.824839 1919 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:14:26.826047 1919 hybrid_clock.cc:648] HybridClock initialized: now 1790435666825997 us; error 34 us; skew 500 ppm
I20260926 15:14:26.826406 1919 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:14:26 dist-test-slave-865v krb5kdc[1906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435666, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:14:26.829574 1919 init.cc:377] Logged in from keytab as kudu/127.12.34.254@KRBTEST.COM (short username kudu)
I20260926 15:14:26.830994 1919 webserver.cc:460] Webserver started at http://127.12.34.254:43007/ using document root <none> and password file <none>
I20260926 15:14:26.831599 1919 fs_manager.cc:359] Metadata directory not provided
I20260926 15:14:26.831696 1919 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:14:26.831933 1919 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:14:26.834748 1919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790435318278681-12427-0/minicluster-data/master-0/data/instance:
uuid: "1e6d7ebadc674e77bcece9767ac39496"
format_stamp: "Formatted at 2026-09-26 15:14:26 on dist-test-slave-865v"
server_key: "c294ecf1c96d3d929a8eb4cf09910299"
server_key_iv: "68ce5f88afd9d8aebaba71240c814985"
server_key_version: "encryptionkey@0"
I20260926 15:14:26.835445 1919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790435318278681-12427-0/minicluster-data/master-0/wal/instance:
uuid: "1e6d7ebadc674e77bcece9767ac39496"
format_stamp: "Formatted at 2026-09-26 15:14:26 on dist-test-slave-865v"
server_key: "c294ecf1c96d3d929a8eb4cf09910299"
server_key_iv: "68ce5f88afd9d8aebaba71240c814985"
server_key_version: "encryptionkey@0"
I20260926 15:14:26.840344 1919 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 15:14:26.843659 1934 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:14:26.845022 1919 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 15:14:26.845269 1919 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790435318278681-12427-0/minicluster-data/master-0/data,/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790435318278681-12427-0/minicluster-data/master-0/wal
uuid: "1e6d7ebadc674e77bcece9767ac39496"
format_stamp: "Formatted at 2026-09-26 15:14:26 on dist-test-slave-865v"
server_key: "c294ecf1c96d3d929a8eb4cf09910299"
server_key_iv: "68ce5f88afd9d8aebaba71240c814985"
server_key_version: "encryptionkey@0"
I20260926 15:14:26.845480 1919 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790435318278681-12427-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790435318278681-12427-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790435318278681-12427-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:14:26.851447 1919 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:14:26.856015 1919 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:14:26.856328 1919 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:14:26.875002 1919 rpc_server.cc:307] RPC server started. Bound to: 127.12.34.254:34943
I20260926 15:14:26.875031 1985 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.34.254:34943 every 8 connection(s)
I20260926 15:14:26.876348 1919 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
I20260926 15:14:26.879971 1986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:14:26.880066 12427 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8uoJGv/build/asan/bin/kudu as pid 1919
I20260926 15:14:26.880280 12427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790435318278681-12427-0/minicluster-data/master-0/wal/instance
I20260926 15:14:26.880722 12427 external_mini_cluster.cc:1428] Setting key e8bec6dbe34717b8b0a49ee523bb28b3
I20260926 15:14:26.889150 1989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:14:26.882547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49217 (local address 127.12.34.254:34943)
0926 15:14:26.883408 (+ 861us) server_negotiation.cc:206] Beginning negotiation
0926 15:14:26.883419 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 15:14:26.883458 (+ 39us) server_negotiation.cc:407] Connection header received
0926 15:14:26.885808 (+ 2350us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:14:26.885829 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:14:26.886153 (+ 324us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:14:26.886475 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:14:26.887596 (+ 1121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:14:26.887602 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:14:26.887614 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:14:26.887627 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:14:26.887683 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:14:26.887707 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:14:26.887865 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0926 15:14:26.888009 (+ 144us) server_negotiation.cc:299] Negotiation successful
0926 15:14:26.888203 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":636,"thread_start_us":406,"threads_started":1}
I20260926 15:14:26.892802 12427 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 15:14:26.892884 1986 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e6d7ebadc674e77bcece9767ac39496: Bootstrap starting.
I20260926 15:14:26.897328 1986 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1e6d7ebadc674e77bcece9767ac39496: Neither blocks nor log segments found. Creating new log.
I20260926 15:14:26.899322 1986 log.cc:824] T 00000000000000000000000000000000 P 1e6d7ebadc674e77bcece9767ac39496: Log is configured to *not* fsync() on all Append() calls
I20260926 15:14:26.902050 1989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:14:26.898948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56287 (local address 127.12.34.254:34943)
0926 15:14:26.899205 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0926 15:14:26.899216 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 15:14:26.899599 (+ 383us) server_negotiation.cc:407] Connection header received
0926 15:14:26.900070 (+ 471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:14:26.900080 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:14:26.900180 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:14:26.900350 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:14:26.901400 (+ 1050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:14:26.901405 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:14:26.901409 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:14:26.901427 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:14:26.901472 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:14:26.901492 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:14:26.901617 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0926 15:14:26.901695 (+ 78us) server_negotiation.cc:299] Negotiation successful
0926 15:14:26.901765 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 15:14:26.902990 1986 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e6d7ebadc674e77bcece9767ac39496: No bootstrap required, opened a new log
I20260926 15:14:26.910234 1986 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1e6d7ebadc674e77bcece9767ac39496 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e6d7ebadc674e77bcece9767ac39496" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 34943 } }
I20260926 15:14:26.910616 1986 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1e6d7ebadc674e77bcece9767ac39496 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:14:26.910722 1986 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1e6d7ebadc674e77bcece9767ac39496 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e6d7ebadc674e77bcece9767ac39496, State: Initialized, Role: FOLLOWER
I20260926 15:14:26.911276 1986 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1e6d7ebadc674e77bcece9767ac39496 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e6d7ebadc674e77bcece9767ac39496" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 34943 } }
I20260926 15:14:26.911525 1986 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1e6d7ebadc674e77bcece9767ac39496 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 15:14:26.911669 1986 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1e6d7ebadc674e77bcece9767ac39496 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 15:14:26.911842 1986 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1e6d7ebadc674e77bcece9767ac39496 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:14:26.914250 1986 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1e6d7ebadc674e77bcece9767ac39496 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e6d7ebadc674e77bcece9767ac39496" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 34943 } }
I20260926 15:14:26.914744 1986 leader_election.cc:302] T 00000000000000000000000000000000 P 1e6d7ebadc674e77bcece9767ac39496 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1e6d7ebadc674e77bcece9767ac39496; no voters:
I20260926 15:14:26.915380 1986 leader_election.cc:288] T 00000000000000000000000000000000 P 1e6d7ebadc674e77bcece9767ac39496 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 15:14:26.915580 1996 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1e6d7ebadc674e77bcece9767ac39496 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:14:26.916317 1996 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1e6d7ebadc674e77bcece9767ac39496 [term 1 LEADER]: Becoming Leader. State: Replica: 1e6d7ebadc674e77bcece9767ac39496, State: Running, Role: LEADER
I20260926 15:14:26.916982 1996 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1e6d7ebadc674e77bcece9767ac39496 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e6d7ebadc674e77bcece9767ac39496" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 34943 } }
I20260926 15:14:26.917734 1986 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1e6d7ebadc674e77bcece9767ac39496 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:14:26.919927 1998 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e6d7ebadc674e77bcece9767ac39496 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1e6d7ebadc674e77bcece9767ac39496. Latest consensus state: current_term: 1 leader_uuid: "1e6d7ebadc674e77bcece9767ac39496" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e6d7ebadc674e77bcece9767ac39496" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 34943 } } }
I20260926 15:14:26.920428 1998 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e6d7ebadc674e77bcece9767ac39496 [sys.catalog]: This master's current role is: LEADER
I20260926 15:14:26.920173 1997 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e6d7ebadc674e77bcece9767ac39496 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1e6d7ebadc674e77bcece9767ac39496" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e6d7ebadc674e77bcece9767ac39496" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 34943 } } }
I20260926 15:14:26.920576 1997 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e6d7ebadc674e77bcece9767ac39496 [sys.catalog]: This master's current role is: LEADER
W20260926 15:14:26.934554 2012 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1e6d7ebadc674e77bcece9767ac39496: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 15:14:26.934698 2012 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 15:14:26.934814 2003 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:14:26.937130 2003 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 15:14:26.946925 2003 catalog_manager.cc:1347] Generated new cluster ID: 27cb1d10ecf542deb45426e2455f0fdf
I20260926 15:14:26.947067 2003 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 15:14:26.976043 2003 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 15:14:26.976480 2003 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:14:26.983199 2003 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1e6d7ebadc674e77bcece9767ac39496: Generated new TSK 0
I20260926 15:14:26.983968 2003 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 15:14:26.992935 12427 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 1919
W20260926 15:14:26.994503 1981 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:49217
W20260926 15:14:26.995253 12427 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 1919: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 15:14:26.995405 12427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8uoJGv/build/asan/bin/kudu with pid 1919
2026-09-26T15:14:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1983 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790435318278681-12427-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 15:14:27 dist-test-slave-865v krb5kdc[2017](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 15:14:27 dist-test-slave-865v krb5kdc[2017](info): set up 2 sockets
Sep 26 15:14:27 dist-test-slave-865v krb5kdc[2017](info): commencing operation
krb5kdc: starting...
W20260926 15:14:28.185827 12427 mini_kdc.cc:121] Time spent starting KDC: real 1.154s user 0.006s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790435318278681-12427-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790435318278681-12427-0/krb5kdc/test-user.keytab.
Sep 26 15:14:28 dist-test-slave-865v krb5kdc[2017](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435668, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T15:14:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T15:14:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.34.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.34.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.34.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.254.keytab.
Entry for principal kudu/127.12.34.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.254.keytab.
I20260926 15:14:28.444468 12427 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8uoJGv/build/asan/bin/kudu
/tmp/dist-test-task8uoJGv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790435318278681-12427-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790435318278681-12427-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790435318278681-12427-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.34.254:33927
--webserver_interface=127.12.34.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.254.keytab
--principal=kudu/127.12.34.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.34.212:44751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.34.254:33927
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790435318278681-12427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790435318278681-12427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790435318278681-12427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:14:28.773602 2030 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:14:28.773953 2030 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:14:28.774119 2030 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:14:28.781507 2030 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:14:28.781628 2030 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:14:28.781677 2030 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:14:28.781729 2030 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:14:28.781770 2030 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:14:28.791611 2030 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.34.212:44751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790435318278681-12427-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790435318278681-12427-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.34.254:33927
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.254.keytab
--principal=kudu/127.12.34.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.34.254:33927
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.34.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790435318278681-12427-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:14:28.792852 2030 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:14:28.794142 2030 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:14:28.805939 2035 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:14:28.806102 2038 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:14:28.806126 2036 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:14:28.806882 2030 server_base.cc:1034] running on GCE node
I20260926 15:14:28.807704 2030 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:14:28.809340 2030 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:14:28.810559 2030 hybrid_clock.cc:648] HybridClock initialized: now 1790435668810506 us; error 49 us; skew 500 ppm
I20260926 15:14:28.810976 2030 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:14:28 dist-test-slave-865v krb5kdc[2017](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435668, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:14:28.820456 2030 init.cc:377] Logged in from keytab as kudu/127.12.34.254@KRBTEST.COM (short username kudu)
I20260926 15:14:28.822266 2030 webserver.cc:460] Webserver started at http://127.12.34.254:45161/ using document root <none> and password file <none>
I20260926 15:14:28.822934 2030 fs_manager.cc:359] Metadata directory not provided
I20260926 15:14:28.823033 2030 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:14:28.823396 2030 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:14:28.826763 2030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790435318278681-12427-0/minicluster-data/master-0/data/instance:
uuid: "834b1c405a3645378c8a6c057e89c866"
format_stamp: "Formatted at 2026-09-26 15:14:28 on dist-test-slave-865v"
server_key: "9cefedb386ab5ee488d833377b5b54f8"
server_key_iv: "c0a3a840546c470b184f06aa1f36cb84"
server_key_version: "encryptionkey@0"
I20260926 15:14:28.827528 2030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790435318278681-12427-0/minicluster-data/master-0/wal/instance:
uuid: "834b1c405a3645378c8a6c057e89c866"
format_stamp: "Formatted at 2026-09-26 15:14:28 on dist-test-slave-865v"
server_key: "9cefedb386ab5ee488d833377b5b54f8"
server_key_iv: "c0a3a840546c470b184f06aa1f36cb84"
server_key_version: "encryptionkey@0"
I20260926 15:14:28.832599 2030 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 15:14:28.835999 2045 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:14:28.837415 2030 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 15:14:28.837601 2030 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790435318278681-12427-0/minicluster-data/master-0/data,/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790435318278681-12427-0/minicluster-data/master-0/wal
uuid: "834b1c405a3645378c8a6c057e89c866"
format_stamp: "Formatted at 2026-09-26 15:14:28 on dist-test-slave-865v"
server_key: "9cefedb386ab5ee488d833377b5b54f8"
server_key_iv: "c0a3a840546c470b184f06aa1f36cb84"
server_key_version: "encryptionkey@0"
I20260926 15:14:28.837795 2030 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790435318278681-12427-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790435318278681-12427-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790435318278681-12427-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:14:28.876348 2030 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:14:28.880712 2030 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:14:28.881018 2030 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:14:28.899672 2030 rpc_server.cc:307] RPC server started. Bound to: 127.12.34.254:33927
I20260926 15:14:28.899691 2096 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.34.254:33927 every 8 connection(s)
I20260926 15:14:28.901067 2030 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
I20260926 15:14:28.904568 2097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:14:28.910640 12427 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8uoJGv/build/asan/bin/kudu as pid 2030
I20260926 15:14:28.910827 12427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790435318278681-12427-0/minicluster-data/master-0/wal/instance
I20260926 15:14:28.911253 12427 external_mini_cluster.cc:1428] Setting key b6c5c799ac8174cea2f2191d51717ed2
I20260926 15:14:28.915253 2097 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 834b1c405a3645378c8a6c057e89c866: Bootstrap starting.
I20260926 15:14:28.918915 2097 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 834b1c405a3645378c8a6c057e89c866: Neither blocks nor log segments found. Creating new log.
I20260926 15:14:28.920049 2097 log.cc:824] T 00000000000000000000000000000000 P 834b1c405a3645378c8a6c057e89c866: Log is configured to *not* fsync() on all Append() calls
I20260926 15:14:28.923435 2097 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 834b1c405a3645378c8a6c057e89c866: No bootstrap required, opened a new log
Sep 26 15:14:28 dist-test-slave-865v krb5kdc[2017](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435668, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.34.254@KRBTEST.COM
I20260926 15:14:28.931937 2097 raft_consensus.cc:348] T 00000000000000000000000000000000 P 834b1c405a3645378c8a6c057e89c866 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "834b1c405a3645378c8a6c057e89c866" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 33927 } }
I20260926 15:14:28.932315 2097 raft_consensus.cc:374] T 00000000000000000000000000000000 P 834b1c405a3645378c8a6c057e89c866 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:14:28.932447 2097 raft_consensus.cc:729] T 00000000000000000000000000000000 P 834b1c405a3645378c8a6c057e89c866 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 834b1c405a3645378c8a6c057e89c866, State: Initialized, Role: FOLLOWER
I20260926 15:14:28.933233 2097 consensus_queue.cc:260] T 00000000000000000000000000000000 P 834b1c405a3645378c8a6c057e89c866 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "834b1c405a3645378c8a6c057e89c866" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 33927 } }
I20260926 15:14:28.933521 2097 raft_consensus.cc:388] T 00000000000000000000000000000000 P 834b1c405a3645378c8a6c057e89c866 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 15:14:28.933658 2097 raft_consensus.cc:482] T 00000000000000000000000000000000 P 834b1c405a3645378c8a6c057e89c866 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 15:14:28.933841 2097 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 834b1c405a3645378c8a6c057e89c866 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:14:28.934808 2100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:14:28.913002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50741 (local address 127.12.34.254:33927)
0926 15:14:28.913865 (+ 863us) server_negotiation.cc:206] Beginning negotiation
0926 15:14:28.913876 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 15:14:28.913917 (+ 41us) server_negotiation.cc:407] Connection header received
0926 15:14:28.915648 (+ 1731us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:14:28.915670 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:14:28.916049 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:14:28.916350 (+ 301us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:14:28.918184 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:14:28.919588 (+ 1404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:14:28.921289 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:14:28.921990 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:14:28.922124 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:14:28.926894 (+ 4770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:14:28.926902 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:14:28.926916 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:14:28.926961 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:14:28.930298 (+ 3337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:14:28.931054 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:14:28.931061 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:14:28.931067 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:14:28.931233 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:14:28.931609 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:14:28.931618 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:14:28.931620 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:14:28.932155 (+ 535us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:14:28.932486 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0926 15:14:28.932886 (+ 400us) server_negotiation.cc:299] Negotiation successful
0926 15:14:28.933384 (+ 498us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":666,"thread_start_us":380,"threads_started":1}
I20260926 15:14:28.936990 2097 raft_consensus.cc:504] T 00000000000000000000000000000000 P 834b1c405a3645378c8a6c057e89c866 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "834b1c405a3645378c8a6c057e89c866" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 33927 } }
I20260926 15:14:28.937597 2097 leader_election.cc:302] T 00000000000000000000000000000000 P 834b1c405a3645378c8a6c057e89c866 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 834b1c405a3645378c8a6c057e89c866; no voters:
I20260926 15:14:28.938244 2097 leader_election.cc:288] T 00000000000000000000000000000000 P 834b1c405a3645378c8a6c057e89c866 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 15:14:28.938439 2102 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 834b1c405a3645378c8a6c057e89c866 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:14:28.938642 12427 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 15:14:28.939043 2102 raft_consensus.cc:686] T 00000000000000000000000000000000 P 834b1c405a3645378c8a6c057e89c866 [term 1 LEADER]: Becoming Leader. State: Replica: 834b1c405a3645378c8a6c057e89c866, State: Running, Role: LEADER
I20260926 15:14:28.939632 2102 consensus_queue.cc:237] T 00000000000000000000000000000000 P 834b1c405a3645378c8a6c057e89c866 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "834b1c405a3645378c8a6c057e89c866" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 33927 } }
I20260926 15:14:28.941359 2102 sys_catalog.cc:455] T 00000000000000000000000000000000 P 834b1c405a3645378c8a6c057e89c866 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 834b1c405a3645378c8a6c057e89c866. Latest consensus state: current_term: 1 leader_uuid: "834b1c405a3645378c8a6c057e89c866" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "834b1c405a3645378c8a6c057e89c866" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 33927 } } }
I20260926 15:14:28.941669 2102 sys_catalog.cc:458] T 00000000000000000000000000000000 P 834b1c405a3645378c8a6c057e89c866 [sys.catalog]: This master's current role is: LEADER
I20260926 15:14:28.942243 2104 sys_catalog.cc:455] T 00000000000000000000000000000000 P 834b1c405a3645378c8a6c057e89c866 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "834b1c405a3645378c8a6c057e89c866" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "834b1c405a3645378c8a6c057e89c866" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 33927 } } }
I20260926 15:14:28.942634 2104 sys_catalog.cc:458] T 00000000000000000000000000000000 P 834b1c405a3645378c8a6c057e89c866 [sys.catalog]: This master's current role is: LEADER
I20260926 15:14:28.950922 2097 sys_catalog.cc:564] T 00000000000000000000000000000000 P 834b1c405a3645378c8a6c057e89c866 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:14:28.951139 2106 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:14:28.956790 2106 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 15:14:28.967720 2106 catalog_manager.cc:1347] Generated new cluster ID: 96292ad22a224286bac1deddba5a6219
I20260926 15:14:28.967841 2106 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 15:14:28.975368 2100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:14:28.951265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58587 (local address 127.12.34.254:33927)
0926 15:14:28.951435 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0926 15:14:28.951441 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:14:28.953740 (+ 2299us) server_negotiation.cc:407] Connection header received
0926 15:14:28.954070 (+ 330us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:14:28.954079 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:14:28.954185 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:14:28.954271 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:14:28.956738 (+ 2467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:14:28.957973 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:14:28.960282 (+ 2309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:14:28.961291 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:14:28.961450 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:14:28.965522 (+ 4072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:14:28.965532 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:14:28.965540 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:14:28.965601 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:14:28.970373 (+ 4772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:14:28.971480 (+ 1107us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:14:28.971489 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:14:28.971496 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:14:28.971679 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:14:28.972223 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:14:28.972231 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:14:28.972237 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:14:28.972791 (+ 554us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:14:28.973104 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0926 15:14:28.974498 (+ 1394us) server_negotiation.cc:299] Negotiation successful
0926 15:14:28.974823 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 15:14:28.980063 2106 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 15:14:28.981966 2106 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:14:28.994115 2106 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 834b1c405a3645378c8a6c057e89c866: Generated new TSK 0
I20260926 15:14:28.994963 2106 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 15:14:29.000193 12427 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 2030
I20260926 15:14:29.001286 2092 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:14:29.091636 12427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8uoJGv/build/asan/bin/kudu with pid 2030
2026-09-26T15:14:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2095 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790435318278681-12427-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 15:14:29 dist-test-slave-865v krb5kdc[2129](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 15:14:29 dist-test-slave-865v krb5kdc[2129](info): set up 2 sockets
Sep 26 15:14:29 dist-test-slave-865v krb5kdc[2129](info): commencing operation
krb5kdc: starting...
W20260926 15:14:30.198222 12427 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.003s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790435318278681-12427-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790435318278681-12427-0/krb5kdc/test-user.keytab.
Sep 26 15:14:30 dist-test-slave-865v krb5kdc[2129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435670, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T15:14:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T15:14:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.34.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.34.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.34.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.254.keytab.
Entry for principal kudu/127.12.34.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.254.keytab.
I20260926 15:14:30.476972 12427 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8uoJGv/build/asan/bin/kudu
/tmp/dist-test-task8uoJGv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790435318278681-12427-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790435318278681-12427-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790435318278681-12427-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.34.254:42833
--webserver_interface=127.12.34.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.254.keytab
--principal=kudu/127.12.34.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.34.212:38403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.34.254:42833
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790435318278681-12427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790435318278681-12427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790435318278681-12427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:14:30.802758 2142 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:14:30.803037 2142 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:14:30.803206 2142 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:14:30.810508 2142 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:14:30.810619 2142 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:14:30.810668 2142 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:14:30.810760 2142 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:14:30.810839 2142 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:14:30.820778 2142 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.34.212:38403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790435318278681-12427-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790435318278681-12427-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.34.254:42833
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.254.keytab
--principal=kudu/127.12.34.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.34.254:42833
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.34.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790435318278681-12427-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:14:30.822124 2142 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:14:30.823427 2142 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:14:30.835242 2150 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:14:30.835265 2148 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:14:30.835500 2147 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:14:30.836035 2142 server_base.cc:1034] running on GCE node
I20260926 15:14:30.836715 2142 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:14:30.838086 2142 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:14:30.839286 2142 hybrid_clock.cc:648] HybridClock initialized: now 1790435670839252 us; error 44 us; skew 500 ppm
I20260926 15:14:30.839620 2142 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:14:30 dist-test-slave-865v krb5kdc[2129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435670, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:14:30.842742 2142 init.cc:377] Logged in from keytab as kudu/127.12.34.254@KRBTEST.COM (short username kudu)
I20260926 15:14:30.844224 2142 webserver.cc:460] Webserver started at http://127.12.34.254:34161/ using document root <none> and password file <none>
I20260926 15:14:30.844905 2142 fs_manager.cc:359] Metadata directory not provided
I20260926 15:14:30.845052 2142 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:14:30.845378 2142 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:14:30.848327 2142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790435318278681-12427-0/minicluster-data/master-0/data/instance:
uuid: "eeff4fe11dd04cdf91f4ff6a8569bd7c"
format_stamp: "Formatted at 2026-09-26 15:14:30 on dist-test-slave-865v"
server_key: "30b5bbfda34f8ec5d01bdd02071c0055"
server_key_iv: "9ff5bbcba6bc9c0bf421299388c8c613"
server_key_version: "encryptionkey@0"
I20260926 15:14:30.849134 2142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790435318278681-12427-0/minicluster-data/master-0/wal/instance:
uuid: "eeff4fe11dd04cdf91f4ff6a8569bd7c"
format_stamp: "Formatted at 2026-09-26 15:14:30 on dist-test-slave-865v"
server_key: "30b5bbfda34f8ec5d01bdd02071c0055"
server_key_iv: "9ff5bbcba6bc9c0bf421299388c8c613"
server_key_version: "encryptionkey@0"
I20260926 15:14:30.854395 2142 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260926 15:14:30.857964 2157 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:14:30.859238 2142 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 15:14:30.859428 2142 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790435318278681-12427-0/minicluster-data/master-0/data,/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790435318278681-12427-0/minicluster-data/master-0/wal
uuid: "eeff4fe11dd04cdf91f4ff6a8569bd7c"
format_stamp: "Formatted at 2026-09-26 15:14:30 on dist-test-slave-865v"
server_key: "30b5bbfda34f8ec5d01bdd02071c0055"
server_key_iv: "9ff5bbcba6bc9c0bf421299388c8c613"
server_key_version: "encryptionkey@0"
I20260926 15:14:30.859627 2142 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790435318278681-12427-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790435318278681-12427-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790435318278681-12427-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:14:30.885159 2142 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:14:30.889685 2142 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:14:30.890000 2142 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:14:30.909891 2142 rpc_server.cc:307] RPC server started. Bound to: 127.12.34.254:42833
I20260926 15:14:30.910006 2208 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.34.254:42833 every 8 connection(s)
I20260926 15:14:30.911373 2142 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
I20260926 15:14:30.912570 12427 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8uoJGv/build/asan/bin/kudu as pid 2142
I20260926 15:14:30.912767 12427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790435318278681-12427-0/minicluster-data/master-0/wal/instance
I20260926 15:14:30.913331 12427 external_mini_cluster.cc:1428] Setting key 1a9f91d78965a4effa31f7282d362a7f
I20260926 15:14:30.916275 2209 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:14:30.926328 2212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:14:30.915147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60285 (local address 127.12.34.254:42833)
0926 15:14:30.915978 (+ 831us) server_negotiation.cc:206] Beginning negotiation
0926 15:14:30.915990 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 15:14:30.916030 (+ 40us) server_negotiation.cc:407] Connection header received
0926 15:14:30.917823 (+ 1793us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:14:30.917844 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:14:30.918216 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:14:30.918599 (+ 383us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:14:30.919144 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:14:30.920412 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:14:30.921775 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:14:30.922414 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:14:30.922611 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:14:30.923843 (+ 1232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:14:30.923853 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:14:30.923866 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:14:30.924420 (+ 554us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:14:30.924500 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:14:30.924530 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:14:30.924664 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0926 15:14:30.924889 (+ 225us) server_negotiation.cc:299] Negotiation successful
0926 15:14:30.925283 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":633,"thread_start_us":356,"threads_started":1}
I20260926 15:14:30.927569 2209 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eeff4fe11dd04cdf91f4ff6a8569bd7c: Bootstrap starting.
I20260926 15:14:30.930595 12427 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 15:14:30.931005 2209 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eeff4fe11dd04cdf91f4ff6a8569bd7c: Neither blocks nor log segments found. Creating new log.
I20260926 15:14:30.931978 2209 log.cc:824] T 00000000000000000000000000000000 P eeff4fe11dd04cdf91f4ff6a8569bd7c: Log is configured to *not* fsync() on all Append() calls
I20260926 15:14:30.935351 2209 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eeff4fe11dd04cdf91f4ff6a8569bd7c: No bootstrap required, opened a new log
I20260926 15:14:30.941886 2212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:14:30.935004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60037 (local address 127.12.34.254:42833)
0926 15:14:30.935257 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0926 15:14:30.935266 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 15:14:30.935845 (+ 579us) server_negotiation.cc:407] Connection header received
0926 15:14:30.936001 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:14:30.936007 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:14:30.936093 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:14:30.936235 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:14:30.936738 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:14:30.937898 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:14:30.939245 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:14:30.939901 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:14:30.940030 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:14:30.941073 (+ 1043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:14:30.941100 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:14:30.941105 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:14:30.941127 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:14:30.941172 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:14:30.941197 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:14:30.941301 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 15:14:30.941429 (+ 128us) server_negotiation.cc:299] Negotiation successful
0926 15:14:30.941532 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 15:14:30.945726 2209 raft_consensus.cc:348] T 00000000000000000000000000000000 P eeff4fe11dd04cdf91f4ff6a8569bd7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eeff4fe11dd04cdf91f4ff6a8569bd7c" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 42833 } }
I20260926 15:14:30.946216 2209 raft_consensus.cc:374] T 00000000000000000000000000000000 P eeff4fe11dd04cdf91f4ff6a8569bd7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:14:30.946355 2209 raft_consensus.cc:729] T 00000000000000000000000000000000 P eeff4fe11dd04cdf91f4ff6a8569bd7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eeff4fe11dd04cdf91f4ff6a8569bd7c, State: Initialized, Role: FOLLOWER
I20260926 15:14:30.946939 2209 consensus_queue.cc:260] T 00000000000000000000000000000000 P eeff4fe11dd04cdf91f4ff6a8569bd7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eeff4fe11dd04cdf91f4ff6a8569bd7c" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 42833 } }
I20260926 15:14:30.947186 2209 raft_consensus.cc:388] T 00000000000000000000000000000000 P eeff4fe11dd04cdf91f4ff6a8569bd7c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 15:14:30.947335 2209 raft_consensus.cc:482] T 00000000000000000000000000000000 P eeff4fe11dd04cdf91f4ff6a8569bd7c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 15:14:30.947517 2209 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eeff4fe11dd04cdf91f4ff6a8569bd7c [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:14:30.949842 2209 raft_consensus.cc:504] T 00000000000000000000000000000000 P eeff4fe11dd04cdf91f4ff6a8569bd7c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eeff4fe11dd04cdf91f4ff6a8569bd7c" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 42833 } }
I20260926 15:14:30.950332 2209 leader_election.cc:302] T 00000000000000000000000000000000 P eeff4fe11dd04cdf91f4ff6a8569bd7c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eeff4fe11dd04cdf91f4ff6a8569bd7c; no voters:
I20260926 15:14:30.951303 2209 leader_election.cc:288] T 00000000000000000000000000000000 P eeff4fe11dd04cdf91f4ff6a8569bd7c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 15:14:30.951339 2219 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eeff4fe11dd04cdf91f4ff6a8569bd7c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:14:30.953060 2219 raft_consensus.cc:686] T 00000000000000000000000000000000 P eeff4fe11dd04cdf91f4ff6a8569bd7c [term 1 LEADER]: Becoming Leader. State: Replica: eeff4fe11dd04cdf91f4ff6a8569bd7c, State: Running, Role: LEADER
I20260926 15:14:30.953635 2219 consensus_queue.cc:237] T 00000000000000000000000000000000 P eeff4fe11dd04cdf91f4ff6a8569bd7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eeff4fe11dd04cdf91f4ff6a8569bd7c" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 42833 } }
I20260926 15:14:30.954013 2209 sys_catalog.cc:564] T 00000000000000000000000000000000 P eeff4fe11dd04cdf91f4ff6a8569bd7c [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:14:30.956274 2221 sys_catalog.cc:455] T 00000000000000000000000000000000 P eeff4fe11dd04cdf91f4ff6a8569bd7c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eeff4fe11dd04cdf91f4ff6a8569bd7c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eeff4fe11dd04cdf91f4ff6a8569bd7c" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 42833 } } }
I20260926 15:14:30.956655 2221 sys_catalog.cc:458] T 00000000000000000000000000000000 P eeff4fe11dd04cdf91f4ff6a8569bd7c [sys.catalog]: This master's current role is: LEADER
I20260926 15:14:30.957283 2220 sys_catalog.cc:455] T 00000000000000000000000000000000 P eeff4fe11dd04cdf91f4ff6a8569bd7c [sys.catalog]: SysCatalogTable state changed. Reason: New leader eeff4fe11dd04cdf91f4ff6a8569bd7c. Latest consensus state: current_term: 1 leader_uuid: "eeff4fe11dd04cdf91f4ff6a8569bd7c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eeff4fe11dd04cdf91f4ff6a8569bd7c" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 42833 } } }
I20260926 15:14:30.957908 2220 sys_catalog.cc:458] T 00000000000000000000000000000000 P eeff4fe11dd04cdf91f4ff6a8569bd7c [sys.catalog]: This master's current role is: LEADER
I20260926 15:14:30.958617 2226 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:14:30.964517 2226 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 15:14:30.975706 2226 catalog_manager.cc:1347] Generated new cluster ID: 68494c1f957644bf87191b18e2df25d3
I20260926 15:14:30.975840 2226 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 15:14:30.985309 2226 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 15:14:30.986692 2226 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:14:30.995437 2226 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eeff4fe11dd04cdf91f4ff6a8569bd7c: Generated new TSK 0
I20260926 15:14:30.996220 2226 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 15:14:31.007210 12427 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 2142
W20260926 15:14:31.008253 2204 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:60285
W20260926 15:14:31.008971 12427 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 2142: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 15:14:31.009147 12427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8uoJGv/build/asan/bin/kudu with pid 2142
2026-09-26T15:14:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1914 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790435318278681-12427-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 15:14:31 dist-test-slave-865v krb5kdc[2241](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 15:14:31 dist-test-slave-865v krb5kdc[2241](info): set up 2 sockets
Sep 26 15:14:31 dist-test-slave-865v krb5kdc[2241](info): commencing operation
krb5kdc: starting...
W20260926 15:14:32.121665 12427 mini_kdc.cc:121] Time spent starting KDC: real 1.080s user 0.010s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790435318278681-12427-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790435318278681-12427-0/krb5kdc/test-user.keytab.
Sep 26 15:14:32 dist-test-slave-865v krb5kdc[2241](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435672, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T15:14:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T15:14:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.34.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.34.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.34.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.254.keytab.
Entry for principal kudu/127.12.34.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.254.keytab.
I20260926 15:14:32.389703 12427 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8uoJGv/build/asan/bin/kudu
/tmp/dist-test-task8uoJGv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790435318278681-12427-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790435318278681-12427-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790435318278681-12427-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.34.254:36141
--webserver_interface=127.12.34.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.254.keytab
--principal=kudu/127.12.34.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.34.212:36293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.34.254:36141
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790435318278681-12427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790435318278681-12427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790435318278681-12427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:14:32.735796 2254 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:14:32.736124 2254 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:14:32.736294 2254 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:14:32.743242 2254 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:14:32.743387 2254 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:14:32.743487 2254 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:14:32.743597 2254 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:14:32.743677 2254 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:14:32.753485 2254 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.34.212:36293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790435318278681-12427-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790435318278681-12427-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.34.254:36141
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.254.keytab
--principal=kudu/127.12.34.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.34.254:36141
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.34.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790435318278681-12427-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:14:32.754679 2254 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:14:32.755920 2254 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:14:32.767474 2262 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:14:32.767541 2260 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:14:32.767575 2259 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:14:32.768447 2254 server_base.cc:1034] running on GCE node
I20260926 15:14:32.769188 2254 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:14:32.770434 2254 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:14:32.771672 2254 hybrid_clock.cc:648] HybridClock initialized: now 1790435672771635 us; error 56 us; skew 500 ppm
I20260926 15:14:32.772028 2254 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:14:32 dist-test-slave-865v krb5kdc[2241](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435672, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:14:32.775130 2254 init.cc:377] Logged in from keytab as kudu/127.12.34.254@KRBTEST.COM (short username kudu)
I20260926 15:14:32.776490 2254 webserver.cc:460] Webserver started at http://127.12.34.254:38667/ using document root <none> and password file <none>
I20260926 15:14:32.777165 2254 fs_manager.cc:359] Metadata directory not provided
I20260926 15:14:32.777268 2254 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:14:32.777522 2254 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:14:32.780277 2254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790435318278681-12427-0/minicluster-data/master-0/data/instance:
uuid: "de2ebbbbe7d2494792e90713c96e77ca"
format_stamp: "Formatted at 2026-09-26 15:14:32 on dist-test-slave-865v"
server_key: "b92bee40cf3675c13dcddbe06bd94f3f"
server_key_iv: "b37bb86bf7856f44602b6ec7a9a77cd5"
server_key_version: "encryptionkey@0"
I20260926 15:14:32.780958 2254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790435318278681-12427-0/minicluster-data/master-0/wal/instance:
uuid: "de2ebbbbe7d2494792e90713c96e77ca"
format_stamp: "Formatted at 2026-09-26 15:14:32 on dist-test-slave-865v"
server_key: "b92bee40cf3675c13dcddbe06bd94f3f"
server_key_iv: "b37bb86bf7856f44602b6ec7a9a77cd5"
server_key_version: "encryptionkey@0"
I20260926 15:14:32.786018 2254 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260926 15:14:32.789625 2269 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:14:32.790920 2254 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260926 15:14:32.791046 2254 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790435318278681-12427-0/minicluster-data/master-0/data,/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790435318278681-12427-0/minicluster-data/master-0/wal
uuid: "de2ebbbbe7d2494792e90713c96e77ca"
format_stamp: "Formatted at 2026-09-26 15:14:32 on dist-test-slave-865v"
server_key: "b92bee40cf3675c13dcddbe06bd94f3f"
server_key_iv: "b37bb86bf7856f44602b6ec7a9a77cd5"
server_key_version: "encryptionkey@0"
I20260926 15:14:32.791206 2254 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790435318278681-12427-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790435318278681-12427-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790435318278681-12427-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:14:32.796789 2254 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:14:32.800729 2254 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:14:32.801005 2254 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:14:32.818902 2254 rpc_server.cc:307] RPC server started. Bound to: 127.12.34.254:36141
I20260926 15:14:32.818945 2320 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.34.254:36141 every 8 connection(s)
I20260926 15:14:32.820220 2254 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
I20260926 15:14:32.823865 2321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:14:32.824115 12427 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8uoJGv/build/asan/bin/kudu as pid 2254
I20260926 15:14:32.824290 12427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790435318278681-12427-0/minicluster-data/master-0/wal/instance
I20260926 15:14:32.824828 12427 external_mini_cluster.cc:1428] Setting key 9301c46ae51c5feb17e7f1ca41f36515
I20260926 15:14:32.833213 2324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:14:32.826740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33521 (local address 127.12.34.254:36141)
0926 15:14:32.827654 (+ 914us) server_negotiation.cc:206] Beginning negotiation
0926 15:14:32.827667 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 15:14:32.827708 (+ 41us) server_negotiation.cc:407] Connection header received
0926 15:14:32.829955 (+ 2247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:14:32.829980 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:14:32.830308 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:14:32.830645 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:14:32.831647 (+ 1002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:14:32.831653 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:14:32.831664 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:14:32.831677 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:14:32.831726 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:14:32.831747 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:14:32.831906 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0926 15:14:32.832005 (+ 99us) server_negotiation.cc:299] Negotiation successful
0926 15:14:32.832205 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":683,"thread_start_us":465,"threads_started":1}
I20260926 15:14:32.836844 12427 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 15:14:32.836989 2321 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de2ebbbbe7d2494792e90713c96e77ca: Bootstrap starting.
I20260926 15:14:32.840838 2321 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P de2ebbbbe7d2494792e90713c96e77ca: Neither blocks nor log segments found. Creating new log.
I20260926 15:14:32.842245 2321 log.cc:824] T 00000000000000000000000000000000 P de2ebbbbe7d2494792e90713c96e77ca: Log is configured to *not* fsync() on all Append() calls
I20260926 15:14:32.845605 2324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:14:32.843071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54137 (local address 127.12.34.254:36141)
0926 15:14:32.843223 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0926 15:14:32.843228 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:14:32.843629 (+ 401us) server_negotiation.cc:407] Connection header received
0926 15:14:32.843818 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:14:32.843821 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:14:32.843868 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:14:32.843945 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:14:32.844912 (+ 967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:14:32.844917 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:14:32.844920 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:14:32.844934 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:14:32.844972 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:14:32.844987 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:14:32.845142 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0926 15:14:32.845252 (+ 110us) server_negotiation.cc:299] Negotiation successful
0926 15:14:32.845298 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260926 15:14:32.845737 2321 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de2ebbbbe7d2494792e90713c96e77ca: No bootstrap required, opened a new log
I20260926 15:14:32.853243 2321 raft_consensus.cc:348] T 00000000000000000000000000000000 P de2ebbbbe7d2494792e90713c96e77ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de2ebbbbe7d2494792e90713c96e77ca" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 36141 } }
I20260926 15:14:32.853561 2321 raft_consensus.cc:374] T 00000000000000000000000000000000 P de2ebbbbe7d2494792e90713c96e77ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:14:32.853706 2321 raft_consensus.cc:729] T 00000000000000000000000000000000 P de2ebbbbe7d2494792e90713c96e77ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de2ebbbbe7d2494792e90713c96e77ca, State: Initialized, Role: FOLLOWER
I20260926 15:14:32.854276 2321 consensus_queue.cc:260] T 00000000000000000000000000000000 P de2ebbbbe7d2494792e90713c96e77ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de2ebbbbe7d2494792e90713c96e77ca" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 36141 } }
I20260926 15:14:32.854578 2321 raft_consensus.cc:388] T 00000000000000000000000000000000 P de2ebbbbe7d2494792e90713c96e77ca [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 15:14:32.854748 2321 raft_consensus.cc:482] T 00000000000000000000000000000000 P de2ebbbbe7d2494792e90713c96e77ca [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 15:14:32.854928 2321 raft_consensus.cc:3037] T 00000000000000000000000000000000 P de2ebbbbe7d2494792e90713c96e77ca [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:14:32.857059 2321 raft_consensus.cc:504] T 00000000000000000000000000000000 P de2ebbbbe7d2494792e90713c96e77ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de2ebbbbe7d2494792e90713c96e77ca" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 36141 } }
I20260926 15:14:32.857542 2321 leader_election.cc:302] T 00000000000000000000000000000000 P de2ebbbbe7d2494792e90713c96e77ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: de2ebbbbe7d2494792e90713c96e77ca; no voters:
I20260926 15:14:32.858067 2321 leader_election.cc:288] T 00000000000000000000000000000000 P de2ebbbbe7d2494792e90713c96e77ca [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 15:14:32.858242 2331 raft_consensus.cc:2781] T 00000000000000000000000000000000 P de2ebbbbe7d2494792e90713c96e77ca [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:14:32.859328 2331 raft_consensus.cc:686] T 00000000000000000000000000000000 P de2ebbbbe7d2494792e90713c96e77ca [term 1 LEADER]: Becoming Leader. State: Replica: de2ebbbbe7d2494792e90713c96e77ca, State: Running, Role: LEADER
I20260926 15:14:32.859804 2331 consensus_queue.cc:237] T 00000000000000000000000000000000 P de2ebbbbe7d2494792e90713c96e77ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de2ebbbbe7d2494792e90713c96e77ca" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 36141 } }
I20260926 15:14:32.860093 2321 sys_catalog.cc:564] T 00000000000000000000000000000000 P de2ebbbbe7d2494792e90713c96e77ca [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:14:32.862360 2333 sys_catalog.cc:455] T 00000000000000000000000000000000 P de2ebbbbe7d2494792e90713c96e77ca [sys.catalog]: SysCatalogTable state changed. Reason: New leader de2ebbbbe7d2494792e90713c96e77ca. Latest consensus state: current_term: 1 leader_uuid: "de2ebbbbe7d2494792e90713c96e77ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de2ebbbbe7d2494792e90713c96e77ca" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 36141 } } }
I20260926 15:14:32.862802 2333 sys_catalog.cc:458] T 00000000000000000000000000000000 P de2ebbbbe7d2494792e90713c96e77ca [sys.catalog]: This master's current role is: LEADER
I20260926 15:14:32.862661 2332 sys_catalog.cc:455] T 00000000000000000000000000000000 P de2ebbbbe7d2494792e90713c96e77ca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "de2ebbbbe7d2494792e90713c96e77ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de2ebbbbe7d2494792e90713c96e77ca" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 36141 } } }
I20260926 15:14:32.862967 2332 sys_catalog.cc:458] T 00000000000000000000000000000000 P de2ebbbbe7d2494792e90713c96e77ca [sys.catalog]: This master's current role is: LEADER
I20260926 15:14:32.863790 2340 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:14:32.869233 2340 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 15:14:32.877658 2340 catalog_manager.cc:1347] Generated new cluster ID: 4ebb8d33ac2e4205a272697f372e5467
I20260926 15:14:32.877784 2340 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 15:14:32.894889 2340 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 15:14:32.895385 2340 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:14:32.907650 2340 catalog_manager.cc:5868] T 00000000000000000000000000000000 P de2ebbbbe7d2494792e90713c96e77ca: Generated new TSK 0
I20260926 15:14:32.908447 2340 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 15:14:32.920471 12427 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 2254
W20260926 15:14:32.921610 2316 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:33521
W20260926 15:14:32.922353 12427 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 2254: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 15:14:32.922499 12427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8uoJGv/build/asan/bin/kudu with pid 2254
2026-09-26T15:14:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1913 ms)
[----------] 18 tests from AuthTokenIssuingTest (35505 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790435318278681-12427-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 15:14:33 dist-test-slave-865v krb5kdc[2352](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 15:14:33 dist-test-slave-865v krb5kdc[2352](info): set up 2 sockets
Sep 26 15:14:33 dist-test-slave-865v krb5kdc[2352](info): commencing operation
krb5kdc: starting...
W20260926 15:14:34.033787 12427 mini_kdc.cc:121] Time spent starting KDC: real 1.078s user 0.002s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790435318278681-12427-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790435318278681-12427-0/krb5kdc/test-user.keytab.
Sep 26 15:14:34 dist-test-slave-865v krb5kdc[2352](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435674, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T15:14:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T15:14:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.34.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.34.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.34.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.254.keytab.
Entry for principal kudu/127.12.34.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.254.keytab.
I20260926 15:14:34.284426 12427 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8uoJGv/build/asan/bin/kudu
/tmp/dist-test-task8uoJGv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790435318278681-12427-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790435318278681-12427-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790435318278681-12427-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.34.254:46301
--webserver_interface=127.12.34.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.254.keytab
--principal=kudu/127.12.34.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.34.212:37321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.34.254:46301,127.12.34.253:40555,127.12.34.252:40371
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790435318278681-12427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790435318278681-12427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790435318278681-12427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:14:34.606688 2365 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 15:14:34.607012 2365 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:14:34.607143 2365 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:14:34.607277 2365 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:14:34.614369 2365 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:14:34.614449 2365 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:14:34.614542 2365 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:14:34.614619 2365 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:14:34.614698 2365 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:14:34.624133 2365 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.34.212:37321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790435318278681-12427-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790435318278681-12427-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.34.254:46301,127.12.34.253:40555,127.12.34.252:40371
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.254.keytab
--principal=kudu/127.12.34.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.34.254:46301
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.34.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790435318278681-12427-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:14:34.625403 2365 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:14:34.626681 2365 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:14:34.637959 2370 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:14:34.638036 2373 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:14:34.638005 2371 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:14:34.638926 2365 server_base.cc:1034] running on GCE node
I20260926 15:14:34.639582 2365 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:14:34.640811 2365 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:14:34.642014 2365 hybrid_clock.cc:648] HybridClock initialized: now 1790435674641939 us; error 58 us; skew 500 ppm
I20260926 15:14:34.642300 2365 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:14:34 dist-test-slave-865v krb5kdc[2352](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435674, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:14:34.658604 2365 init.cc:377] Logged in from keytab as kudu/127.12.34.254@KRBTEST.COM (short username kudu)
I20260926 15:14:34.660140 2365 webserver.cc:460] Webserver started at http://127.12.34.254:46471/ using document root <none> and password file <none>
I20260926 15:14:34.660806 2365 fs_manager.cc:359] Metadata directory not provided
I20260926 15:14:34.660893 2365 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:14:34.661221 2365 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:14:34.664069 2365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790435318278681-12427-0/minicluster-data/master-0/data/instance:
uuid: "0e3f86c440934790aeb23b0ae70b3ebb"
format_stamp: "Formatted at 2026-09-26 15:14:34 on dist-test-slave-865v"
server_key: "44bc87cc1b77801e1d9e8ed7dbb44817"
server_key_iv: "7424965aa0cc5a0c60b7d7290a2789b8"
server_key_version: "encryptionkey@0"
I20260926 15:14:34.664798 2365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790435318278681-12427-0/minicluster-data/master-0/wal/instance:
uuid: "0e3f86c440934790aeb23b0ae70b3ebb"
format_stamp: "Formatted at 2026-09-26 15:14:34 on dist-test-slave-865v"
server_key: "44bc87cc1b77801e1d9e8ed7dbb44817"
server_key_iv: "7424965aa0cc5a0c60b7d7290a2789b8"
server_key_version: "encryptionkey@0"
I20260926 15:14:34.669857 2365 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 15:14:34.673996 2380 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:14:34.675391 2365 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 15:14:34.675540 2365 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790435318278681-12427-0/minicluster-data/master-0/data,/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790435318278681-12427-0/minicluster-data/master-0/wal
uuid: "0e3f86c440934790aeb23b0ae70b3ebb"
format_stamp: "Formatted at 2026-09-26 15:14:34 on dist-test-slave-865v"
server_key: "44bc87cc1b77801e1d9e8ed7dbb44817"
server_key_iv: "7424965aa0cc5a0c60b7d7290a2789b8"
server_key_version: "encryptionkey@0"
I20260926 15:14:34.675740 2365 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790435318278681-12427-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790435318278681-12427-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790435318278681-12427-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:14:34.692499 2365 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:14:34.696797 2365 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:14:34.697104 2365 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:14:34.715689 2365 rpc_server.cc:307] RPC server started. Bound to: 127.12.34.254:46301
I20260926 15:14:34.715777 2431 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.34.254:46301 every 8 connection(s)
I20260926 15:14:34.717034 2365 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
I20260926 15:14:34.717718 12427 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8uoJGv/build/asan/bin/kudu as pid 2365
I20260926 15:14:34.717958 12427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790435318278681-12427-0/minicluster-data/master-0/wal/instance
I20260926 15:14:34.718719 12427 external_mini_cluster.cc:1428] Setting key 6e96ade6315daa3437b4a4fdf19e623d
I20260926 15:14:34.720980 2432 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:14:34.727471 2432 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.34.254" port: 46301 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.12.34.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.34.253@KRBTEST.COM" created.
Sep 26 15:14:34 dist-test-slave-865v krb5kdc[2352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435674, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.254@KRBTEST.COM for kudu/127.12.34.254@KRBTEST.COM
I20260926 15:14:34.755508 2435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:14:34.731250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:46301 (local address 127.0.0.1:53008)
0926 15:14:34.732185 (+ 935us) negotiation.cc:107] Waiting for socket to connect
0926 15:14:34.732238 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 15:14:34.735024 (+ 2786us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:14:34.735920 (+ 896us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:14:34.735929 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:14:34.736735 (+ 806us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:14:34.737226 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:14:34.737251 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:14:34.738909 (+ 1658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:14:34.738918 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:14:34.740900 (+ 1982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:14:34.740920 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:14:34.741898 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:14:34.741908 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:14:34.742115 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:14:34.742796 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:14:34.742828 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:14:34.748047 (+ 5219us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:14:34.751519 (+ 3472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:14:34.751535 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:14:34.751548 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:14:34.751928 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:14:34.752386 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:14:34.752390 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:14:34.752393 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:14:34.752567 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:14:34.753404 (+ 837us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:14:34.753411 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:14:34.753600 (+ 189us) client_negotiation.cc:769] Sending connection context
0926 15:14:34.753871 (+ 271us) client_negotiation.cc:240] Negotiation successful
0926 15:14:34.754359 (+ 488us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":642,"thread_start_us":426,"threads_started":1}
I20260926 15:14:34.756418 2436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:14:34.731250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53008 (local address 127.12.34.254:46301)
0926 15:14:34.733189 (+ 1939us) server_negotiation.cc:206] Beginning negotiation
0926 15:14:34.733195 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:14:34.733248 (+ 53us) server_negotiation.cc:407] Connection header received
0926 15:14:34.735448 (+ 2200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:14:34.735453 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:14:34.735679 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:14:34.735843 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:14:34.737438 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:14:34.738749 (+ 1311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:14:34.741057 (+ 2308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:14:34.741638 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:14:34.741768 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:14:34.748296 (+ 6528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:14:34.748305 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:14:34.748318 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:14:34.748350 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:14:34.751298 (+ 2948us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:14:34.752072 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:14:34.752080 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:14:34.752083 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:14:34.752229 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:14:34.752684 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:14:34.752689 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:14:34.752692 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:14:34.753256 (+ 564us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:14:34.753465 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0926 15:14:34.755710 (+ 2245us) server_negotiation.cc:299] Negotiation successful
0926 15:14:34.755963 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1686,"spinlock_wait_cycles":14592,"thread_start_us":1471,"threads_started":1}
I20260926 15:14:34.758953 2432 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.34.253" port: 40555 } has no permanent_uuid. Determining permanent_uuid...
I20260926 15:14:34.760099 2435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:14:34.759577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.253:40555 (local address 127.0.0.1:38414)
0926 15:14:34.759814 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0926 15:14:34.759942 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.253:40555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260926 15:14:34.760341 2381 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.34.253:40555: connect: Connection refused (error 111)
I20260926 15:14:34.761986 2435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:14:34.761549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.253:40555 (local address 127.0.0.1:38430)
0926 15:14:34.761724 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0926 15:14:34.761816 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.253:40555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260926 15:14:34.762202 2432 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.34.253:40555: Network error: Client connection negotiation failed: client connection to 127.12.34.253:40555: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.34.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.253.keytab.
Entry for principal kudu/127.12.34.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.253.keytab.
I20260926 15:14:34.782001 12427 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8uoJGv/build/asan/bin/kudu
/tmp/dist-test-task8uoJGv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790435318278681-12427-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790435318278681-12427-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790435318278681-12427-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790435318278681-12427-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.34.253:40555
--webserver_interface=127.12.34.253
--webserver_port=0
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.253.keytab
--principal=kudu/127.12.34.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.34.212:37321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.34.254:46301,127.12.34.253:40555,127.12.34.252:40371
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790435318278681-12427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790435318278681-12427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790435318278681-12427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 15:14:34.811564 2432 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.34.253" port: 40555 } attempt: 1
I20260926 15:14:34.812783 2435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:14:34.812270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.253:40555 (local address 127.0.0.1:38434)
0926 15:14:34.812492 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0926 15:14:34.812614 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.253:40555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260926 15:14:34.813789 2435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:14:34.813402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.253:40555 (local address 127.0.0.1:38442)
0926 15:14:34.813556 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 15:14:34.813637 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.253:40555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260926 15:14:34.813974 2432 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.34.253:40555: Network error: Client connection negotiation failed: client connection to 127.12.34.253:40555: connect: Connection refused (error 111)
I20260926 15:14:34.882233 2432 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.34.253" port: 40555 } attempt: 2
I20260926 15:14:34.883473 2435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:14:34.882910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.253:40555 (local address 127.0.0.1:38452)
0926 15:14:34.883190 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0926 15:14:34.883307 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.253:40555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260926 15:14:34.884500 2435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:14:34.884108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.253:40555 (local address 127.0.0.1:38466)
0926 15:14:34.884266 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0926 15:14:34.884359 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.253:40555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260926 15:14:34.884708 2432 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.34.253:40555: Network error: Client connection negotiation failed: client connection to 127.12.34.253:40555: connect: Connection refused (error 111)
I20260926 15:14:34.975992 2432 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.34.253" port: 40555 } attempt: 3
I20260926 15:14:34.977285 2435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:14:34.976719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.253:40555 (local address 127.0.0.1:38482)
0926 15:14:34.976972 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0926 15:14:34.977110 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.253:40555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
I20260926 15:14:34.978364 2435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:14:34.977948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.253:40555 (local address 127.0.0.1:38484)
0926 15:14:34.978113 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0926 15:14:34.978210 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.253:40555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260926 15:14:34.978600 2432 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.34.253:40555: Network error: Client connection negotiation failed: client connection to 127.12.34.253:40555: connect: Connection refused (error 111)
W20260926 15:14:35.116954 2439 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 15:14:35.117332 2439 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:14:35.117468 2439 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:14:35.117583 2439 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260926 15:14:35.121878 2432 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.34.253" port: 40555 } attempt: 4
I20260926 15:14:35.123178 2435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:14:35.122632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.253:40555 (local address 127.0.0.1:38498)
0926 15:14:35.122947 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0926 15:14:35.123038 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.253:40555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164}
I20260926 15:14:35.124591 2435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:14:35.124145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.253:40555 (local address 127.0.0.1:38500)
0926 15:14:35.124332 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0926 15:14:35.124427 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.253:40555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
W20260926 15:14:35.124835 2432 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.34.253:40555: Network error: Client connection negotiation failed: client connection to 127.12.34.253:40555: connect: Connection refused (error 111)
W20260926 15:14:35.125826 2439 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:14:35.125908 2439 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:14:35.125990 2439 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:14:35.126062 2439 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:14:35.126120 2439 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:14:35.136143 2439 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.34.212:37321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790435318278681-12427-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790435318278681-12427-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.12.34.254:46301,127.12.34.253:40555,127.12.34.252:40371
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.253.keytab
--principal=kudu/127.12.34.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.34.253:40555
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790435318278681-12427-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.12.34.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790435318278681-12427-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:14:35.137544 2439 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:14:35.138831 2439 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:14:35.151059 2444 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:14:35.151132 2445 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:14:35.151028 2447 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:14:35.155121 2439 server_base.cc:1034] running on GCE node
I20260926 15:14:35.156189 2439 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:14:35.157859 2439 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:14:35.159144 2439 hybrid_clock.cc:648] HybridClock initialized: now 1790435675159087 us; error 28 us; skew 500 ppm
I20260926 15:14:35.159507 2439 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:14:35 dist-test-slave-865v krb5kdc[2352](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435675, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:14:35.163990 2439 init.cc:377] Logged in from keytab as kudu/127.12.34.253@KRBTEST.COM (short username kudu)
I20260926 15:14:35.165825 2439 webserver.cc:460] Webserver started at http://127.12.34.253:39511/ using document root <none> and password file <none>
I20260926 15:14:35.166568 2439 fs_manager.cc:359] Metadata directory not provided
I20260926 15:14:35.166710 2439 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:14:35.167030 2439 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:14:35.170099 2439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790435318278681-12427-0/minicluster-data/master-1/data/instance:
uuid: "5a1918de22234eac8d9214ca9afff005"
format_stamp: "Formatted at 2026-09-26 15:14:35 on dist-test-slave-865v"
server_key: "b733793de7162c15f7e1fe527d638a61"
server_key_iv: "2e367e3e4c599a02b98af93b943f929f"
server_key_version: "encryptionkey@0"
I20260926 15:14:35.170917 2439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790435318278681-12427-0/minicluster-data/master-1/wal/instance:
uuid: "5a1918de22234eac8d9214ca9afff005"
format_stamp: "Formatted at 2026-09-26 15:14:35 on dist-test-slave-865v"
server_key: "b733793de7162c15f7e1fe527d638a61"
server_key_iv: "2e367e3e4c599a02b98af93b943f929f"
server_key_version: "encryptionkey@0"
I20260926 15:14:35.177034 2439 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260926 15:14:35.180899 2454 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:14:35.182303 2439 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 15:14:35.182499 2439 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790435318278681-12427-0/minicluster-data/master-1/data,/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790435318278681-12427-0/minicluster-data/master-1/wal
uuid: "5a1918de22234eac8d9214ca9afff005"
format_stamp: "Formatted at 2026-09-26 15:14:35 on dist-test-slave-865v"
server_key: "b733793de7162c15f7e1fe527d638a61"
server_key_iv: "2e367e3e4c599a02b98af93b943f929f"
server_key_version: "encryptionkey@0"
I20260926 15:14:35.182727 2439 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790435318278681-12427-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790435318278681-12427-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790435318278681-12427-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:14:35.207909 2439 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:14:35.212776 2439 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:14:35.213097 2439 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:14:35.232477 2439 rpc_server.cc:307] RPC server started. Bound to: 127.12.34.253:40555
I20260926 15:14:35.232514 2505 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.34.253:40555 every 8 connection(s)
I20260926 15:14:35.233976 2439 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790435318278681-12427-0/minicluster-data/master-1/data/info.pb
I20260926 15:14:35.237562 2506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:14:35.238706 12427 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8uoJGv/build/asan/bin/kudu as pid 2439
I20260926 15:14:35.238883 12427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790435318278681-12427-0/minicluster-data/master-1/wal/instance
I20260926 15:14:35.239389 12427 external_mini_cluster.cc:1428] Setting key 9d195317cd3c063fddcbd4785749a04b
I20260926 15:14:35.247360 2506 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.34.254" port: 46301 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 15:14:35 dist-test-slave-865v krb5kdc[2352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435675, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.253@KRBTEST.COM for kudu/127.12.34.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.12.34.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.34.252@KRBTEST.COM" created.
I20260926 15:14:35.271288 2436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:14:35.250743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53022 (local address 127.12.34.254:46301)
0926 15:14:35.250922 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0926 15:14:35.250926 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:14:35.252379 (+ 1453us) server_negotiation.cc:407] Connection header received
0926 15:14:35.254465 (+ 2086us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:14:35.254475 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:14:35.254586 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:14:35.254719 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:14:35.256218 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:14:35.257533 (+ 1315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:14:35.259066 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:14:35.259889 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:14:35.260051 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:14:35.263530 (+ 3479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:14:35.263538 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:14:35.263544 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:14:35.263590 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:14:35.267817 (+ 4227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:14:35.268677 (+ 860us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:14:35.268681 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:14:35.268683 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:14:35.268973 (+ 290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:14:35.269507 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:14:35.269511 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:14:35.269513 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:14:35.269817 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:14:35.269931 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0926 15:14:35.270767 (+ 836us) server_negotiation.cc:299] Negotiation successful
0926 15:14:35.271034 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 15:14:35.272567 2509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:14:35.251421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:46301 (local address 127.0.0.1:53022)
0926 15:14:35.252215 (+ 794us) negotiation.cc:107] Waiting for socket to connect
0926 15:14:35.252251 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 15:14:35.254097 (+ 1846us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:14:35.254924 (+ 827us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:14:35.254932 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:14:35.255497 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:14:35.256036 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:14:35.256052 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:14:35.257708 (+ 1656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:14:35.257712 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:14:35.258889 (+ 1177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:14:35.258902 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:14:35.260093 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:14:35.260097 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:14:35.260284 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:14:35.260989 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:14:35.261011 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:14:35.263287 (+ 2276us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:14:35.267999 (+ 4712us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:14:35.268006 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:14:35.268020 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:14:35.268544 (+ 524us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:14:35.269130 (+ 586us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:14:35.269135 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:14:35.269138 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:14:35.269389 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:14:35.269936 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:14:35.269942 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:14:35.270327 (+ 385us) client_negotiation.cc:769] Sending connection context
0926 15:14:35.270563 (+ 236us) client_negotiation.cc:240] Negotiation successful
0926 15:14:35.271111 (+ 548us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":349,"threads_started":1}
I20260926 15:14:35.274326 2506 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.34.253" port: 40555 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 15:14:35 dist-test-slave-865v krb5kdc[2352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435675, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.253@KRBTEST.COM for kudu/127.12.34.253@KRBTEST.COM
I20260926 15:14:35.292165 2509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:14:35.274981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.253:40555 (local address 127.0.0.1:38506)
0926 15:14:35.275172 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0926 15:14:35.275196 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 15:14:35.275297 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:14:35.276479 (+ 1182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:14:35.276483 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:14:35.276863 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:14:35.277010 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:14:35.277018 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:14:35.279081 (+ 2063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:14:35.279085 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:14:35.280202 (+ 1117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:14:35.280220 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:14:35.281023 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:14:35.281035 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:14:35.281267 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:14:35.281879 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:14:35.281907 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:14:35.285006 (+ 3099us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:14:35.289751 (+ 4745us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:14:35.289755 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:14:35.289757 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:14:35.290143 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:14:35.290568 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:14:35.290571 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:14:35.290573 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:14:35.290694 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:14:35.291447 (+ 753us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:14:35.291451 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:14:35.291577 (+ 126us) client_negotiation.cc:769] Sending connection context
0926 15:14:35.291696 (+ 119us) client_negotiation.cc:240] Negotiation successful
0926 15:14:35.291898 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260926 15:14:35.292654 2510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:14:35.275278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38506 (local address 127.12.34.253:40555)
0926 15:14:35.275979 (+ 701us) server_negotiation.cc:206] Beginning negotiation
0926 15:14:35.275990 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 15:14:35.276024 (+ 34us) server_negotiation.cc:407] Connection header received
0926 15:14:35.276078 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:14:35.276088 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:14:35.276305 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:14:35.276452 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:14:35.277226 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:14:35.278923 (+ 1697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:14:35.280349 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:14:35.280867 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:14:35.280963 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:14:35.285292 (+ 4329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:14:35.285309 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:14:35.285330 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:14:35.285371 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:14:35.289604 (+ 4233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:14:35.290281 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:14:35.290292 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:14:35.290299 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:14:35.290443 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:14:35.290816 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:14:35.290823 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:14:35.290829 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:14:35.291309 (+ 480us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:14:35.291520 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0926 15:14:35.291885 (+ 365us) server_negotiation.cc:299] Negotiation successful
0926 15:14:35.292181 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":526,"thread_start_us":372,"threads_started":1}
I20260926 15:14:35.295075 2506 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.34.252" port: 40371 } has no permanent_uuid. Determining permanent_uuid...
I20260926 15:14:35.296532 2509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:14:35.295910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.252:40371 (local address 127.0.0.1:32930)
0926 15:14:35.296141 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0926 15:14:35.296319 (+ 178us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.252:40371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
W20260926 15:14:35.296854 2457 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.34.252:40371: connect: Connection refused (error 111)
I20260926 15:14:35.298647 2509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:14:35.298170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.252:40371 (local address 127.0.0.1:32934)
0926 15:14:35.298388 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0926 15:14:35.298499 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.252:40371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260926 15:14:35.298924 2506 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.34.252:40371: Network error: Client connection negotiation failed: client connection to 127.12.34.252:40371: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.34.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.252.keytab.
Entry for principal kudu/127.12.34.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.252.keytab.
I20260926 15:14:35.309275 12427 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8uoJGv/build/asan/bin/kudu
/tmp/dist-test-task8uoJGv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790435318278681-12427-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790435318278681-12427-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790435318278681-12427-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790435318278681-12427-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.34.252:40371
--webserver_interface=127.12.34.252
--webserver_port=0
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.252.keytab
--principal=kudu/127.12.34.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.34.212:37321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.34.254:46301,127.12.34.253:40555,127.12.34.252:40371
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790435318278681-12427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790435318278681-12427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790435318278681-12427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 15:14:35.348318 2506 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.34.252" port: 40371 } attempt: 1
I20260926 15:14:35.349572 2509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:14:35.348999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.252:40371 (local address 127.0.0.1:32946)
0926 15:14:35.349243 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0926 15:14:35.349395 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.252:40371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260926 15:14:35.350611 2509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:14:35.350229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.252:40371 (local address 127.0.0.1:32962)
0926 15:14:35.350406 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0926 15:14:35.350493 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.252:40371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260926 15:14:35.350853 2506 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.34.252:40371: Network error: Client connection negotiation failed: client connection to 127.12.34.252:40371: connect: Connection refused (error 111)
I20260926 15:14:35.419184 2506 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.34.252" port: 40371 } attempt: 2
I20260926 15:14:35.420442 2509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:14:35.419900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.252:40371 (local address 127.0.0.1:32964)
0926 15:14:35.420128 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0926 15:14:35.420251 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.252:40371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260926 15:14:35.421553 2509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:14:35.421199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.252:40371 (local address 127.0.0.1:32974)
0926 15:14:35.421345 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0926 15:14:35.421423 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.252:40371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260926 15:14:35.421799 2506 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.34.252:40371: Network error: Client connection negotiation failed: client connection to 127.12.34.252:40371: connect: Connection refused (error 111)
I20260926 15:14:35.424105 2432 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.34.253" port: 40555 } attempt: 5
Sep 26 15:14:35 dist-test-slave-865v krb5kdc[2352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435674, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.254@KRBTEST.COM for kudu/127.12.34.253@KRBTEST.COM
I20260926 15:14:35.437616 2435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:14:35.424709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.253:40555 (local address 127.0.0.1:38512)
0926 15:14:35.424903 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0926 15:14:35.424922 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 15:14:35.425071 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:14:35.425479 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:14:35.425485 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:14:35.425955 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:14:35.426188 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:14:35.426201 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:14:35.427177 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:14:35.427182 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:14:35.428511 (+ 1329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:14:35.428527 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:14:35.429353 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:14:35.429359 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:14:35.429572 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:14:35.430175 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:14:35.430198 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:14:35.432506 (+ 2308us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:14:35.435419 (+ 2913us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:14:35.435426 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:14:35.435430 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:14:35.435805 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:14:35.436170 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:14:35.436177 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:14:35.436180 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:14:35.436321 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:14:35.436842 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:14:35.436847 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:14:35.436999 (+ 152us) client_negotiation.cc:769] Sending connection context
0926 15:14:35.437177 (+ 178us) client_negotiation.cc:240] Negotiation successful
0926 15:14:35.437382 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260926 15:14:35.437819 2510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:14:35.424779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38512 (local address 127.12.34.253:40555)
0926 15:14:35.424934 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0926 15:14:35.424938 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:14:35.425003 (+ 65us) server_negotiation.cc:407] Connection header received
0926 15:14:35.425242 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:14:35.425246 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:14:35.425322 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:14:35.425401 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:14:35.426343 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:14:35.427044 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:14:35.428660 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:14:35.429215 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:14:35.429310 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:14:35.432701 (+ 3391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:14:35.432705 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:14:35.432708 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:14:35.432737 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:14:35.435278 (+ 2541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:14:35.435930 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:14:35.435938 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:14:35.435941 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:14:35.436055 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:14:35.436427 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:14:35.436434 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:14:35.436437 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:14:35.436715 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:14:35.436875 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0926 15:14:35.437314 (+ 439us) server_negotiation.cc:299] Negotiation successful
0926 15:14:35.437527 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 15:14:35.438867 2432 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.34.252" port: 40371 } has no permanent_uuid. Determining permanent_uuid...
I20260926 15:14:35.440003 2435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:14:35.439647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.252:40371 (local address 127.0.0.1:32986)
0926 15:14:35.439804 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0926 15:14:35.439883 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.252:40371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260926 15:14:35.440907 2435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:14:35.440592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.252:40371 (local address 127.0.0.1:32996)
0926 15:14:35.440743 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0926 15:14:35.440807 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.252:40371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260926 15:14:35.441120 2432 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.34.252:40371: Network error: Client connection negotiation failed: client connection to 127.12.34.252:40371: connect: Connection refused (error 111)
I20260926 15:14:35.492355 2432 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.34.252" port: 40371 } attempt: 1
I20260926 15:14:35.493495 2435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:14:35.493003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.252:40371 (local address 127.0.0.1:32998)
0926 15:14:35.493236 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0926 15:14:35.493330 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.252:40371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260926 15:14:35.494450 2435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:14:35.494137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.252:40371 (local address 127.0.0.1:33006)
0926 15:14:35.494281 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0926 15:14:35.494353 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.252:40371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260926 15:14:35.494645 2432 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.34.252:40371: Network error: Client connection negotiation failed: client connection to 127.12.34.252:40371: connect: Connection refused (error 111)
I20260926 15:14:35.513024 2506 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.34.252" port: 40371 } attempt: 3
I20260926 15:14:35.514204 2509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:14:35.513689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.252:40371 (local address 127.0.0.1:33014)
0926 15:14:35.513914 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0926 15:14:35.514021 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.252:40371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260926 15:14:35.515398 2509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:14:35.514889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.252:40371 (local address 127.0.0.1:33020)
0926 15:14:35.515121 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0926 15:14:35.515246 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.252:40371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131}
W20260926 15:14:35.515618 2506 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.34.252:40371: Network error: Client connection negotiation failed: client connection to 127.12.34.252:40371: connect: Connection refused (error 111)
I20260926 15:14:35.562860 2432 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.34.252" port: 40371 } attempt: 2
I20260926 15:14:35.564025 2435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:14:35.563508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.252:40371 (local address 127.0.0.1:33026)
0926 15:14:35.563706 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0926 15:14:35.563852 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.252:40371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260926 15:14:35.564981 2435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:14:35.564629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.252:40371 (local address 127.0.0.1:33034)
0926 15:14:35.564783 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 15:14:35.564868 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.252:40371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260926 15:14:35.565210 2432 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.34.252:40371: Network error: Client connection negotiation failed: client connection to 127.12.34.252:40371: connect: Connection refused (error 111)
W20260926 15:14:35.647003 2513 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 15:14:35.647383 2513 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:14:35.647506 2513 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:14:35.647640 2513 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:14:35.655234 2513 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:14:35.655370 2513 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:14:35.655493 2513 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:14:35.655872 2513 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:14:35.656086 2513 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:14:35.658864 2506 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.34.252" port: 40371 } attempt: 4
I20260926 15:14:35.660151 2509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:14:35.659555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.252:40371 (local address 127.0.0.1:33048)
0926 15:14:35.659854 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0926 15:14:35.659960 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.252:40371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260926 15:14:35.661235 2509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:14:35.660844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.252:40371 (local address 127.0.0.1:33054)
0926 15:14:35.660994 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0926 15:14:35.661112 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.252:40371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260926 15:14:35.661463 2506 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.34.252:40371: Network error: Client connection negotiation failed: client connection to 127.12.34.252:40371: connect: Connection refused (error 111)
I20260926 15:14:35.667461 2513 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.34.212:37321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790435318278681-12427-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790435318278681-12427-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.12.34.254:46301,127.12.34.253:40555,127.12.34.252:40371
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.252.keytab
--principal=kudu/127.12.34.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.34.252:40371
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790435318278681-12427-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.12.34.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790435318278681-12427-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:14:35.668785 2513 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:14:35.670132 2513 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 15:14:35.671423 2432 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.34.252" port: 40371 } attempt: 3
I20260926 15:14:35.672513 2435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:14:35.672053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.252:40371 (local address 127.0.0.1:33062)
0926 15:14:35.672286 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0926 15:14:35.672367 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.252:40371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260926 15:14:35.673585 2435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:14:35.673201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.252:40371 (local address 127.0.0.1:33068)
0926 15:14:35.673421 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0926 15:14:35.673491 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.252:40371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
W20260926 15:14:35.673815 2432 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.34.252:40371: Network error: Client connection negotiation failed: client connection to 127.12.34.252:40371: connect: Connection refused (error 111)
W20260926 15:14:35.682329 2518 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:14:35.682359 2521 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:14:35.682392 2519 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:14:35.683456 2513 server_base.cc:1034] running on GCE node
I20260926 15:14:35.684201 2513 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:14:35.685719 2513 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:14:35.686990 2513 hybrid_clock.cc:648] HybridClock initialized: now 1790435675686924 us; error 35 us; skew 500 ppm
I20260926 15:14:35.687342 2513 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:14:35 dist-test-slave-865v krb5kdc[2352](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435675, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:14:35.690565 2513 init.cc:377] Logged in from keytab as kudu/127.12.34.252@KRBTEST.COM (short username kudu)
I20260926 15:14:35.692123 2513 webserver.cc:460] Webserver started at http://127.12.34.252:44615/ using document root <none> and password file <none>
I20260926 15:14:35.692818 2513 fs_manager.cc:359] Metadata directory not provided
I20260926 15:14:35.692957 2513 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:14:35.693333 2513 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:14:35.696224 2513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790435318278681-12427-0/minicluster-data/master-2/data/instance:
uuid: "9cd1628c92d34b9a84fb83656c4961b4"
format_stamp: "Formatted at 2026-09-26 15:14:35 on dist-test-slave-865v"
server_key: "837f07289a154aecf8bd1bb8477306c3"
server_key_iv: "dcfd5616281a4180eb8db6736937e791"
server_key_version: "encryptionkey@0"
I20260926 15:14:35.697017 2513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790435318278681-12427-0/minicluster-data/master-2/wal/instance:
uuid: "9cd1628c92d34b9a84fb83656c4961b4"
format_stamp: "Formatted at 2026-09-26 15:14:35 on dist-test-slave-865v"
server_key: "837f07289a154aecf8bd1bb8477306c3"
server_key_iv: "dcfd5616281a4180eb8db6736937e791"
server_key_version: "encryptionkey@0"
I20260926 15:14:35.702206 2513 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 15:14:35.706312 2528 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:14:35.707876 2513 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260926 15:14:35.708123 2513 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790435318278681-12427-0/minicluster-data/master-2/data,/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790435318278681-12427-0/minicluster-data/master-2/wal
uuid: "9cd1628c92d34b9a84fb83656c4961b4"
format_stamp: "Formatted at 2026-09-26 15:14:35 on dist-test-slave-865v"
server_key: "837f07289a154aecf8bd1bb8477306c3"
server_key_iv: "dcfd5616281a4180eb8db6736937e791"
server_key_version: "encryptionkey@0"
I20260926 15:14:35.708380 2513 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790435318278681-12427-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790435318278681-12427-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790435318278681-12427-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:14:35.734879 2513 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:14:35.739383 2513 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:14:35.739671 2513 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:14:35.759009 2513 rpc_server.cc:307] RPC server started. Bound to: 127.12.34.252:40371
I20260926 15:14:35.759063 2579 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.34.252:40371 every 8 connection(s)
I20260926 15:14:35.760362 2513 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790435318278681-12427-0/minicluster-data/master-2/data/info.pb
I20260926 15:14:35.764039 2580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:14:35.767324 12427 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8uoJGv/build/asan/bin/kudu as pid 2513
I20260926 15:14:35.767499 12427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790435318278681-12427-0/minicluster-data/master-2/wal/instance
I20260926 15:14:35.767956 12427 external_mini_cluster.cc:1428] Setting key a9552d02b03f60c6d29731926d592ce9
I20260926 15:14:35.771142 2580 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.34.254" port: 46301 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 15:14:35 dist-test-slave-865v krb5kdc[2352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435674, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.34.252@KRBTEST.COM
Sep 26 15:14:35 dist-test-slave-865v krb5kdc[2352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435675, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.252@KRBTEST.COM for kudu/127.12.34.254@KRBTEST.COM
I20260926 15:14:35.791643 2584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:14:35.774592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53032 (local address 127.12.34.254:46301)
0926 15:14:35.775203 (+ 611us) server_negotiation.cc:206] Beginning negotiation
0926 15:14:35.775212 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 15:14:35.775915 (+ 703us) server_negotiation.cc:407] Connection header received
0926 15:14:35.776077 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:14:35.776082 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:14:35.776164 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:14:35.776296 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:14:35.777737 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:14:35.778885 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:14:35.780556 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:14:35.780964 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:14:35.781126 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:14:35.785193 (+ 4067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:14:35.785201 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:14:35.785207 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:14:35.785253 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:14:35.788554 (+ 3301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:14:35.789429 (+ 875us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:14:35.789434 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:14:35.789437 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:14:35.789583 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:14:35.789984 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:14:35.789989 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:14:35.789994 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:14:35.790369 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:14:35.790677 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0926 15:14:35.791010 (+ 333us) server_negotiation.cc:299] Negotiation successful
0926 15:14:35.791237 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":439,"thread_start_us":316,"threads_started":1}
I20260926 15:14:35.792124 2583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:14:35.769858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33074 (local address 127.12.34.252:40371)
0926 15:14:35.770608 (+ 750us) server_negotiation.cc:206] Beginning negotiation
0926 15:14:35.770619 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 15:14:35.770657 (+ 38us) server_negotiation.cc:407] Connection header received
0926 15:14:35.772212 (+ 1555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:14:35.772232 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:14:35.772617 (+ 385us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:14:35.772996 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:14:35.774599 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:14:35.776237 (+ 1638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:14:35.777952 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:14:35.778924 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:14:35.779103 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:14:35.783939 (+ 4836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:14:35.783953 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:14:35.783973 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:14:35.784015 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:14:35.787680 (+ 3665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:14:35.788380 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:14:35.788389 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:14:35.788400 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:14:35.788593 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:14:35.789030 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:14:35.789035 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:14:35.789038 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:14:35.789769 (+ 731us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:14:35.790060 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0926 15:14:35.790437 (+ 377us) server_negotiation.cc:299] Negotiation successful
0926 15:14:35.790870 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":55,"server-negotiator.queue_time_us":555,"thread_start_us":350,"threads_started":1}
I20260926 15:14:35.792171 2585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:14:35.774598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:46301 (local address 127.0.0.1:53032)
0926 15:14:35.775752 (+ 1154us) negotiation.cc:107] Waiting for socket to connect
0926 15:14:35.775805 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 15:14:35.775956 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:14:35.776385 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:14:35.776391 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:14:35.777279 (+ 888us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:14:35.777577 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:14:35.777589 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:14:35.779073 (+ 1484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:14:35.779078 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:14:35.780410 (+ 1332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:14:35.780422 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:14:35.781213 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:14:35.781220 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:14:35.781372 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:14:35.782217 (+ 845us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:14:35.782260 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:14:35.784939 (+ 2679us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:14:35.788723 (+ 3784us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:14:35.788735 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:14:35.788742 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:14:35.789313 (+ 571us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:14:35.789695 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:14:35.789698 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:14:35.789700 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:14:35.789850 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:14:35.790527 (+ 677us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:14:35.790543 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:14:35.790723 (+ 180us) client_negotiation.cc:769] Sending connection context
0926 15:14:35.790856 (+ 133us) client_negotiation.cc:240] Negotiation successful
0926 15:14:35.791076 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":927,"mutex_wait_us":3,"thread_start_us":792,"threads_started":1}
I20260926 15:14:35.793705 2580 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.34.253" port: 40555 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 15:14:35 dist-test-slave-865v krb5kdc[2352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435675, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.252@KRBTEST.COM for kudu/127.12.34.253@KRBTEST.COM
Sep 26 15:14:35 dist-test-slave-865v krb5kdc[2352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435674, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.34.254@KRBTEST.COM
I20260926 15:14:35.807801 2585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:14:35.794322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.253:40555 (local address 127.0.0.1:38518)
0926 15:14:35.794487 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0926 15:14:35.794506 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 15:14:35.794600 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:14:35.795189 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:14:35.795192 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:14:35.795502 (+ 310us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:14:35.795634 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:14:35.795640 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:14:35.797002 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:14:35.797006 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:14:35.797988 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:14:35.798000 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:14:35.798925 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:14:35.798929 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:14:35.799081 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:14:35.799628 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:14:35.799644 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:14:35.801961 (+ 2317us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:14:35.805458 (+ 3497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:14:35.805463 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:14:35.805467 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:14:35.805818 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:14:35.806212 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:14:35.806217 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:14:35.806221 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:14:35.806359 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:14:35.806874 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:14:35.806879 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:14:35.807033 (+ 154us) client_negotiation.cc:769] Sending connection context
0926 15:14:35.807184 (+ 151us) client_negotiation.cc:240] Negotiation successful
0926 15:14:35.807445 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260926 15:14:35.807830 2510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:14:35.794545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38518 (local address 127.12.34.253:40555)
0926 15:14:35.794742 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0926 15:14:35.794750 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 15:14:35.794770 (+ 20us) server_negotiation.cc:407] Connection header received
0926 15:14:35.794834 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:14:35.794839 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:14:35.794917 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:14:35.795012 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:14:35.795755 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:14:35.796855 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:14:35.798123 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:14:35.798771 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:14:35.798896 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:14:35.802166 (+ 3270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:14:35.802174 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:14:35.802182 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:14:35.802229 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:14:35.805320 (+ 3091us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:14:35.805939 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:14:35.805946 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:14:35.805949 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:14:35.806093 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:14:35.806472 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:14:35.806477 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:14:35.806480 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:14:35.806747 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:14:35.806912 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0926 15:14:35.807271 (+ 359us) server_negotiation.cc:299] Negotiation successful
0926 15:14:35.807471 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 15:14:35.808876 2580 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.34.252" port: 40371 } has no permanent_uuid. Determining permanent_uuid...
I20260926 15:14:35.809692 2584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:14:35.795063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53034 (local address 127.12.34.254:46301)
0926 15:14:35.795221 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0926 15:14:35.795227 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:14:35.795243 (+ 16us) server_negotiation.cc:407] Connection header received
0926 15:14:35.795371 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:14:35.795376 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:14:35.795448 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:14:35.795549 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:14:35.797178 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:14:35.797837 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:14:35.799389 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:14:35.799804 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:14:35.799945 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:14:35.804261 (+ 4316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:14:35.804271 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:14:35.804274 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:14:35.804303 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:14:35.806926 (+ 2623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:14:35.807577 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:14:35.807587 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:14:35.807590 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:14:35.807698 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:14:35.808141 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:14:35.808150 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:14:35.808155 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:14:35.808500 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:14:35.808788 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0926 15:14:35.809199 (+ 411us) server_negotiation.cc:299] Negotiation successful
0926 15:14:35.809358 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
Sep 26 15:14:35 dist-test-slave-865v krb5kdc[2352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435675, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.252@KRBTEST.COM for kudu/127.12.34.252@KRBTEST.COM
Sep 26 15:14:35 dist-test-slave-865v krb5kdc[2352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435674, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.34.253@KRBTEST.COM
I20260926 15:14:35.823223 2583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:14:35.809803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33088 (local address 127.12.34.252:40371)
0926 15:14:35.810198 (+ 395us) server_negotiation.cc:206] Beginning negotiation
0926 15:14:35.810203 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:14:35.810251 (+ 48us) server_negotiation.cc:407] Connection header received
0926 15:14:35.810289 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:14:35.810292 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:14:35.810343 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:14:35.810408 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:14:35.811255 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:14:35.812205 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:14:35.813574 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:14:35.814132 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:14:35.814224 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:14:35.817130 (+ 2906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:14:35.817139 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:14:35.817147 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:14:35.817187 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:14:35.820422 (+ 3235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:14:35.821208 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:14:35.821214 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:14:35.821220 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:14:35.821375 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:14:35.821730 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:14:35.821735 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:14:35.821739 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:14:35.822036 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:14:35.822144 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 15:14:35.822655 (+ 511us) server_negotiation.cc:299] Negotiation successful
0926 15:14:35.822971 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":99,"server-negotiator.queue_time_us":286}
I20260926 15:14:35.823303 2585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:14:35.809907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.252:40371 (local address 127.0.0.1:33088)
0926 15:14:35.810057 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0926 15:14:35.810074 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 15:14:35.810176 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:14:35.810467 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:14:35.810472 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:14:35.810925 (+ 453us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:14:35.811051 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:14:35.811060 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:14:35.812336 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:14:35.812339 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:14:35.813443 (+ 1104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:14:35.813455 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:14:35.814296 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:14:35.814302 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:14:35.814518 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:14:35.815051 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:14:35.815075 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:14:35.816921 (+ 1846us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:14:35.820575 (+ 3654us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:14:35.820581 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:14:35.820585 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:14:35.821047 (+ 462us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:14:35.821488 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:14:35.821491 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:14:35.821494 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:14:35.821619 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:14:35.822154 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:14:35.822159 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:14:35.822312 (+ 153us) client_negotiation.cc:769] Sending connection context
0926 15:14:35.822505 (+ 193us) client_negotiation.cc:240] Negotiation successful
0926 15:14:35.822830 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260926 15:14:35.827522 2510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:14:35.811445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38522 (local address 127.12.34.253:40555)
0926 15:14:35.812382 (+ 937us) server_negotiation.cc:206] Beginning negotiation
0926 15:14:35.812386 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:14:35.812398 (+ 12us) server_negotiation.cc:407] Connection header received
0926 15:14:35.812432 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:14:35.812435 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:14:35.812484 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:14:35.812597 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:14:35.813957 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:14:35.814907 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:14:35.816099 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:14:35.816567 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:14:35.816709 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:14:35.821908 (+ 5199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:14:35.821912 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:14:35.821916 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:14:35.821946 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:14:35.824952 (+ 3006us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:14:35.825744 (+ 792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:14:35.825748 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:14:35.825750 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:14:35.825892 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:14:35.826290 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:14:35.826294 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:14:35.826296 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:14:35.826560 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:14:35.826782 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0926 15:14:35.827079 (+ 297us) server_negotiation.cc:299] Negotiation successful
0926 15:14:35.827244 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":849}
I20260926 15:14:35.829310 2580 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9cd1628c92d34b9a84fb83656c4961b4: Bootstrap starting.
I20260926 15:14:35.832176 12427 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 15:14:35.833279 2580 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9cd1628c92d34b9a84fb83656c4961b4: Neither blocks nor log segments found. Creating new log.
I20260926 15:14:35.834600 2580 log.cc:824] T 00000000000000000000000000000000 P 9cd1628c92d34b9a84fb83656c4961b4: Log is configured to *not* fsync() on all Append() calls
I20260926 15:14:35.837287 2580 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9cd1628c92d34b9a84fb83656c4961b4: No bootstrap required, opened a new log
I20260926 15:14:35.835754 2407 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "0e3f86c440934790aeb23b0ae70b3ebb"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53034
I20260926 15:14:35.844214 2580 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9cd1628c92d34b9a84fb83656c4961b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e3f86c440934790aeb23b0ae70b3ebb" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 46301 } } peers { permanent_uuid: "5a1918de22234eac8d9214ca9afff005" member_type: VOTER last_known_addr { host: "127.12.34.253" port: 40555 } } peers { permanent_uuid: "9cd1628c92d34b9a84fb83656c4961b4" member_type: VOTER last_known_addr { host: "127.12.34.252" port: 40371 } }
I20260926 15:14:35.844544 2580 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9cd1628c92d34b9a84fb83656c4961b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cd1628c92d34b9a84fb83656c4961b4, State: Initialized, Role: FOLLOWER
I20260926 15:14:35.844982 2580 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9cd1628c92d34b9a84fb83656c4961b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e3f86c440934790aeb23b0ae70b3ebb" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 46301 } } peers { permanent_uuid: "5a1918de22234eac8d9214ca9afff005" member_type: VOTER last_known_addr { host: "127.12.34.253" port: 40555 } } peers { permanent_uuid: "9cd1628c92d34b9a84fb83656c4961b4" member_type: VOTER last_known_addr { host: "127.12.34.252" port: 40371 } }
I20260926 15:14:35.845954 2587 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9cd1628c92d34b9a84fb83656c4961b4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e3f86c440934790aeb23b0ae70b3ebb" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 46301 } } peers { permanent_uuid: "5a1918de22234eac8d9214ca9afff005" member_type: VOTER last_known_addr { host: "127.12.34.253" port: 40555 } } peers { permanent_uuid: "9cd1628c92d34b9a84fb83656c4961b4" member_type: VOTER last_known_addr { host: "127.12.34.252" port: 40371 } } }
I20260926 15:14:35.846352 2587 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9cd1628c92d34b9a84fb83656c4961b4 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 15:14:35.847052 2580 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9cd1628c92d34b9a84fb83656c4961b4 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:14:35.851011 2432 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.34.252" port: 40371 } attempt: 4
Sep 26 15:14:35 dist-test-slave-865v krb5kdc[2352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435674, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.254@KRBTEST.COM for kudu/127.12.34.252@KRBTEST.COM
I20260926 15:14:35.854120 2513 master_runner.cc:186] Error getting master registration for 127.12.34.254:46301: OK, instance_id { permanent_uuid: "0e3f86c440934790aeb23b0ae70b3ebb" instance_seqno: 1790435674710425 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 15:14:35.863583 2513 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260926 15:14:35.867606 2598 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9cd1628c92d34b9a84fb83656c4961b4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 15:14:35.867735 2598 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 15:14:35.869940 2435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:14:35.851757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.252:40371 (local address 127.0.0.1:33094)
0926 15:14:35.851976 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0926 15:14:35.851998 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 15:14:35.852124 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:14:35.853070 (+ 946us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:14:35.853101 (+ 31us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:14:35.853622 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:14:35.853835 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:14:35.853847 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:14:35.855421 (+ 1574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:14:35.855429 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:14:35.857403 (+ 1974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:14:35.857424 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:14:35.858769 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:14:35.858775 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:14:35.858976 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:14:35.859786 (+ 810us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:14:35.859816 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:14:35.862710 (+ 2894us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:14:35.866879 (+ 4169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:14:35.866885 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:14:35.866889 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:14:35.867389 (+ 500us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:14:35.867895 (+ 506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:14:35.867900 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:14:35.867904 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:14:35.868081 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:14:35.868881 (+ 800us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:14:35.868888 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:14:35.869052 (+ 164us) client_negotiation.cc:769] Sending connection context
0926 15:14:35.869231 (+ 179us) client_negotiation.cc:240] Negotiation successful
0926 15:14:35.869529 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260926 15:14:35.869951 2583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:14:35.852462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33094 (local address 127.12.34.252:40371)
0926 15:14:35.852638 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0926 15:14:35.852644 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:14:35.852665 (+ 21us) server_negotiation.cc:407] Connection header received
0926 15:14:35.852727 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:14:35.852732 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:14:35.852801 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:14:35.852898 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:14:35.853995 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:14:35.855227 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:14:35.857570 (+ 2343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:14:35.858415 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:14:35.858540 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:14:35.862896 (+ 4356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:14:35.862903 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:14:35.862908 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:14:35.862954 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:14:35.866719 (+ 3765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:14:35.867530 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:14:35.867536 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:14:35.867540 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:14:35.867750 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:14:35.868251 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:14:35.868258 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:14:35.868263 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:14:35.868730 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:14:35.868894 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0926 15:14:35.869354 (+ 460us) server_negotiation.cc:299] Negotiation successful
0926 15:14:35.869578 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 15:14:35.874909 2432 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e3f86c440934790aeb23b0ae70b3ebb: Bootstrap starting.
I20260926 15:14:35.878510 2432 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0e3f86c440934790aeb23b0ae70b3ebb: Neither blocks nor log segments found. Creating new log.
I20260926 15:14:35.879675 2432 log.cc:824] T 00000000000000000000000000000000 P 0e3f86c440934790aeb23b0ae70b3ebb: Log is configured to *not* fsync() on all Append() calls
I20260926 15:14:35.882468 2432 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e3f86c440934790aeb23b0ae70b3ebb: No bootstrap required, opened a new log
I20260926 15:14:35.888562 2432 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0e3f86c440934790aeb23b0ae70b3ebb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e3f86c440934790aeb23b0ae70b3ebb" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 46301 } } peers { permanent_uuid: "5a1918de22234eac8d9214ca9afff005" member_type: VOTER last_known_addr { host: "127.12.34.253" port: 40555 } } peers { permanent_uuid: "9cd1628c92d34b9a84fb83656c4961b4" member_type: VOTER last_known_addr { host: "127.12.34.252" port: 40371 } }
I20260926 15:14:35.888952 2432 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0e3f86c440934790aeb23b0ae70b3ebb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e3f86c440934790aeb23b0ae70b3ebb, State: Initialized, Role: FOLLOWER
I20260926 15:14:35.889484 2432 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0e3f86c440934790aeb23b0ae70b3ebb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e3f86c440934790aeb23b0ae70b3ebb" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 46301 } } peers { permanent_uuid: "5a1918de22234eac8d9214ca9afff005" member_type: VOTER last_known_addr { host: "127.12.34.253" port: 40555 } } peers { permanent_uuid: "9cd1628c92d34b9a84fb83656c4961b4" member_type: VOTER last_known_addr { host: "127.12.34.252" port: 40371 } }
I20260926 15:14:35.890666 2601 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e3f86c440934790aeb23b0ae70b3ebb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e3f86c440934790aeb23b0ae70b3ebb" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 46301 } } peers { permanent_uuid: "5a1918de22234eac8d9214ca9afff005" member_type: VOTER last_known_addr { host: "127.12.34.253" port: 40555 } } peers { permanent_uuid: "9cd1628c92d34b9a84fb83656c4961b4" member_type: VOTER last_known_addr { host: "127.12.34.252" port: 40371 } } }
I20260926 15:14:35.891337 2601 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e3f86c440934790aeb23b0ae70b3ebb [sys.catalog]: This master's current role is: FOLLOWER
I20260926 15:14:35.891939 2432 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0e3f86c440934790aeb23b0ae70b3ebb [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:14:35.891973 2407 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0e3f86c440934790aeb23b0ae70b3ebb [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260926 15:14:35.892144 2407 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0e3f86c440934790aeb23b0ae70b3ebb [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:14:35.896028 2407 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0e3f86c440934790aeb23b0ae70b3ebb [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e3f86c440934790aeb23b0ae70b3ebb" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 46301 } } peers { permanent_uuid: "5a1918de22234eac8d9214ca9afff005" member_type: VOTER last_known_addr { host: "127.12.34.253" port: 40555 } } peers { permanent_uuid: "9cd1628c92d34b9a84fb83656c4961b4" member_type: VOTER last_known_addr { host: "127.12.34.252" port: 40371 } }
I20260926 15:14:35.898564 2612 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 15:14:35.899106 2555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0e3f86c440934790aeb23b0ae70b3ebb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "9cd1628c92d34b9a84fb83656c4961b4"
I20260926 15:14:35.899650 2555 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9cd1628c92d34b9a84fb83656c4961b4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:14:35.901633 2407 leader_election.cc:288] T 00000000000000000000000000000000 P 0e3f86c440934790aeb23b0ae70b3ebb [CANDIDATE]: Term 1 election: Requested vote from peers 5a1918de22234eac8d9214ca9afff005 (127.12.34.253:40555), 9cd1628c92d34b9a84fb83656c4961b4 (127.12.34.252:40371)
I20260926 15:14:35.903287 2555 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9cd1628c92d34b9a84fb83656c4961b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e3f86c440934790aeb23b0ae70b3ebb in term 1.
I20260926 15:14:35.904021 2383 leader_election.cc:302] T 00000000000000000000000000000000 P 0e3f86c440934790aeb23b0ae70b3ebb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e3f86c440934790aeb23b0ae70b3ebb, 9cd1628c92d34b9a84fb83656c4961b4; no voters:
I20260926 15:14:35.904528 2601 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0e3f86c440934790aeb23b0ae70b3ebb [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:14:35.914440 2601 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0e3f86c440934790aeb23b0ae70b3ebb [term 1 LEADER]: Becoming Leader. State: Replica: 0e3f86c440934790aeb23b0ae70b3ebb, State: Running, Role: LEADER
I20260926 15:14:35.915238 2601 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0e3f86c440934790aeb23b0ae70b3ebb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e3f86c440934790aeb23b0ae70b3ebb" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 46301 } } peers { permanent_uuid: "5a1918de22234eac8d9214ca9afff005" member_type: VOTER last_known_addr { host: "127.12.34.253" port: 40555 } } peers { permanent_uuid: "9cd1628c92d34b9a84fb83656c4961b4" member_type: VOTER last_known_addr { host: "127.12.34.252" port: 40371 } }
I20260926 15:14:35.903545 2365 master_runner.cc:186] Error getting master registration for 127.12.34.253:40555: OK, instance_id { permanent_uuid: "5a1918de22234eac8d9214ca9afff005" instance_seqno: 1790435675227074 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 15:14:35.919965 2365 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 15:14:35.924021 2616 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e3f86c440934790aeb23b0ae70b3ebb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0e3f86c440934790aeb23b0ae70b3ebb. Latest consensus state: current_term: 1 leader_uuid: "0e3f86c440934790aeb23b0ae70b3ebb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e3f86c440934790aeb23b0ae70b3ebb" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 46301 } } peers { permanent_uuid: "5a1918de22234eac8d9214ca9afff005" member_type: VOTER last_known_addr { host: "127.12.34.253" port: 40555 } } peers { permanent_uuid: "9cd1628c92d34b9a84fb83656c4961b4" member_type: VOTER last_known_addr { host: "127.12.34.252" port: 40371 } } }
I20260926 15:14:35.924525 2616 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e3f86c440934790aeb23b0ae70b3ebb [sys.catalog]: This master's current role is: LEADER
I20260926 15:14:35.925478 2622 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:14:35.937463 2583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:14:35.907653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33098 (local address 127.12.34.252:40371)
0926 15:14:35.907825 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0926 15:14:35.907831 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:14:35.914964 (+ 7133us) server_negotiation.cc:407] Connection header received
0926 15:14:35.916219 (+ 1255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:14:35.916226 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:14:35.916341 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:14:35.916419 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:14:35.919584 (+ 3165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:14:35.920845 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:14:35.923015 (+ 2170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:14:35.923722 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:14:35.923844 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:14:35.927994 (+ 4150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:14:35.928003 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:14:35.928010 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:14:35.928062 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:14:35.931851 (+ 3789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:14:35.933213 (+ 1362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:14:35.933246 (+ 33us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:14:35.933250 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:14:35.933508 (+ 258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:14:35.934334 (+ 826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:14:35.934339 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:14:35.934369 (+ 30us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:14:35.935055 (+ 686us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:14:35.935470 (+ 415us) server_negotiation.cc:1035] Waiting for connection context
0926 15:14:35.936400 (+ 930us) server_negotiation.cc:299] Negotiation successful
0926 15:14:35.936808 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 15:14:35.944258 2584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:14:35.907392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53046 (local address 127.12.34.254:46301)
0926 15:14:35.907652 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0926 15:14:35.907662 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 15:14:35.920368 (+ 12706us) server_negotiation.cc:407] Connection header received
0926 15:14:35.920657 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:14:35.920663 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:14:35.920744 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:14:35.920862 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:14:35.925390 (+ 4528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:14:35.926494 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:14:35.929057 (+ 2563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:14:35.930003 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:14:35.930123 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:14:35.934351 (+ 4228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:14:35.934362 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:14:35.934369 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:14:35.934424 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:14:35.939111 (+ 4687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:14:35.940472 (+ 1361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:14:35.940480 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:14:35.940487 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:14:35.940696 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:14:35.941326 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:14:35.941334 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:14:35.941340 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:14:35.941809 (+ 469us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:14:35.942137 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0926 15:14:35.943215 (+ 1078us) server_negotiation.cc:299] Negotiation successful
0926 15:14:35.943513 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260926 15:14:35.949420 2622 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260926 15:14:35.949903 2386 tablet.cc:2369] T 00000000000000000000000000000000 P 0e3f86c440934790aeb23b0ae70b3ebb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 15:14:35.954326 2510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:14:35.915501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38530 (local address 127.12.34.253:40555)
0926 15:14:35.915719 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0926 15:14:35.915725 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:14:35.925863 (+ 10138us) server_negotiation.cc:407] Connection header received
0926 15:14:35.926358 (+ 495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:14:35.926366 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:14:35.926464 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:14:35.926567 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:14:35.928857 (+ 2290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:14:35.930046 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:14:35.932047 (+ 2001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:14:35.932726 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:14:35.932825 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:14:35.939599 (+ 6774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:14:35.939606 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:14:35.939610 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:14:35.939657 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:14:35.943622 (+ 3965us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:14:35.944529 (+ 907us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:14:35.944537 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:14:35.944542 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:14:35.944704 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:14:35.945150 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:14:35.945167 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:14:35.945171 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:14:35.945516 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:14:35.945763 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0926 15:14:35.953703 (+ 7940us) server_negotiation.cc:299] Negotiation successful
0926 15:14:35.953960 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260926 15:14:35.960631 2506 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.34.252" port: 40371 } attempt: 5
I20260926 15:14:35.960834 2555 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9cd1628c92d34b9a84fb83656c4961b4 [term 1 FOLLOWER]: Refusing update from remote peer 0e3f86c440934790aeb23b0ae70b3ebb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:14:35.962427 2601 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0e3f86c440934790aeb23b0ae70b3ebb [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cd1628c92d34b9a84fb83656c4961b4" member_type: VOTER last_known_addr { host: "127.12.34.252" port: 40371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Sep 26 15:14:35 dist-test-slave-865v krb5kdc[2352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435675, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.253@KRBTEST.COM for kudu/127.12.34.252@KRBTEST.COM
I20260926 15:14:35.988178 2509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:14:35.961175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.252:40371 (local address 127.0.0.1:33112)
0926 15:14:35.961408 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0926 15:14:35.961426 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 15:14:35.961527 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:14:35.961948 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:14:35.961953 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:14:35.962454 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:14:35.962709 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:14:35.962722 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:14:35.968545 (+ 5823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:14:35.968551 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:14:35.970010 (+ 1459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:14:35.970028 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:14:35.971294 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:14:35.971300 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:14:35.971552 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:14:35.972327 (+ 775us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:14:35.972352 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:14:35.975188 (+ 2836us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:14:35.982451 (+ 7263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:14:35.982483 (+ 32us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:14:35.982488 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:14:35.983448 (+ 960us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:14:35.984327 (+ 879us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:14:35.984333 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:14:35.984337 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:14:35.984637 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:14:35.986159 (+ 1522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:14:35.986164 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:14:35.986482 (+ 318us) client_negotiation.cc:769] Sending connection context
0926 15:14:35.986836 (+ 354us) client_negotiation.cc:240] Negotiation successful
0926 15:14:35.987494 (+ 658us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120}
I20260926 15:14:35.990434 2583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:14:35.961403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33112 (local address 127.12.34.252:40371)
0926 15:14:35.961668 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0926 15:14:35.961674 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:14:35.961692 (+ 18us) server_negotiation.cc:407] Connection header received
0926 15:14:35.961746 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:14:35.961751 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:14:35.961820 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:14:35.961920 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:14:35.963200 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:14:35.964305 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:14:35.970197 (+ 5892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:14:35.970935 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:14:35.971062 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:14:35.975439 (+ 4377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:14:35.975446 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:14:35.975456 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:14:35.975492 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:14:35.982202 (+ 6710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:14:35.983774 (+ 1572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:14:35.983780 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:14:35.983784 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:14:35.984103 (+ 319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:14:35.984919 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:14:35.984923 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:14:35.984926 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:14:35.985902 (+ 976us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:14:35.988796 (+ 2894us) server_negotiation.cc:1035] Waiting for connection context
0926 15:14:35.989126 (+ 330us) server_negotiation.cc:299] Negotiation successful
0926 15:14:35.989609 (+ 483us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142}
I20260926 15:14:35.990831 2587 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9cd1628c92d34b9a84fb83656c4961b4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0e3f86c440934790aeb23b0ae70b3ebb. Latest consensus state: current_term: 1 leader_uuid: "0e3f86c440934790aeb23b0ae70b3ebb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e3f86c440934790aeb23b0ae70b3ebb" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 46301 } } peers { permanent_uuid: "5a1918de22234eac8d9214ca9afff005" member_type: VOTER last_known_addr { host: "127.12.34.253" port: 40555 } } peers { permanent_uuid: "9cd1628c92d34b9a84fb83656c4961b4" member_type: VOTER last_known_addr { host: "127.12.34.252" port: 40371 } } }
I20260926 15:14:35.991142 2587 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9cd1628c92d34b9a84fb83656c4961b4 [sys.catalog]: This master's current role is: FOLLOWER
W20260926 15:14:35.992024 2534 tablet.cc:2369] T 00000000000000000000000000000000 P 9cd1628c92d34b9a84fb83656c4961b4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 15:14:35.996843 2506 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a1918de22234eac8d9214ca9afff005: Bootstrap starting.
I20260926 15:14:36.001896 2616 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e3f86c440934790aeb23b0ae70b3ebb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0e3f86c440934790aeb23b0ae70b3ebb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e3f86c440934790aeb23b0ae70b3ebb" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 46301 } } peers { permanent_uuid: "5a1918de22234eac8d9214ca9afff005" member_type: VOTER last_known_addr { host: "127.12.34.253" port: 40555 } } peers { permanent_uuid: "9cd1628c92d34b9a84fb83656c4961b4" member_type: VOTER last_known_addr { host: "127.12.34.252" port: 40371 } } }
I20260926 15:14:36.002177 2616 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e3f86c440934790aeb23b0ae70b3ebb [sys.catalog]: This master's current role is: LEADER
I20260926 15:14:36.004719 2622 catalog_manager.cc:1347] Generated new cluster ID: 276162dc3fbd4b83b92b7d74e5f94540
I20260926 15:14:36.004866 2622 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 15:14:36.007009 2506 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5a1918de22234eac8d9214ca9afff005: Neither blocks nor log segments found. Creating new log.
I20260926 15:14:36.008196 2506 log.cc:824] T 00000000000000000000000000000000 P 5a1918de22234eac8d9214ca9afff005: Log is configured to *not* fsync() on all Append() calls
I20260926 15:14:36.012723 2587 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9cd1628c92d34b9a84fb83656c4961b4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0e3f86c440934790aeb23b0ae70b3ebb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e3f86c440934790aeb23b0ae70b3ebb" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 46301 } } peers { permanent_uuid: "5a1918de22234eac8d9214ca9afff005" member_type: VOTER last_known_addr { host: "127.12.34.253" port: 40555 } } peers { permanent_uuid: "9cd1628c92d34b9a84fb83656c4961b4" member_type: VOTER last_known_addr { host: "127.12.34.252" port: 40371 } } }
I20260926 15:14:36.012954 2587 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9cd1628c92d34b9a84fb83656c4961b4 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 15:14:36.022732 2506 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a1918de22234eac8d9214ca9afff005: No bootstrap required, opened a new log
I20260926 15:14:36.038415 2506 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5a1918de22234eac8d9214ca9afff005 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e3f86c440934790aeb23b0ae70b3ebb" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 46301 } } peers { permanent_uuid: "5a1918de22234eac8d9214ca9afff005" member_type: VOTER last_known_addr { host: "127.12.34.253" port: 40555 } } peers { permanent_uuid: "9cd1628c92d34b9a84fb83656c4961b4" member_type: VOTER last_known_addr { host: "127.12.34.252" port: 40371 } }
I20260926 15:14:36.039255 2506 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5a1918de22234eac8d9214ca9afff005 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a1918de22234eac8d9214ca9afff005, State: Initialized, Role: FOLLOWER
I20260926 15:14:36.040081 2506 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5a1918de22234eac8d9214ca9afff005 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e3f86c440934790aeb23b0ae70b3ebb" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 46301 } } peers { permanent_uuid: "5a1918de22234eac8d9214ca9afff005" member_type: VOTER last_known_addr { host: "127.12.34.253" port: 40555 } } peers { permanent_uuid: "9cd1628c92d34b9a84fb83656c4961b4" member_type: VOTER last_known_addr { host: "127.12.34.252" port: 40371 } }
I20260926 15:14:36.053699 2630 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a1918de22234eac8d9214ca9afff005 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e3f86c440934790aeb23b0ae70b3ebb" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 46301 } } peers { permanent_uuid: "5a1918de22234eac8d9214ca9afff005" member_type: VOTER last_known_addr { host: "127.12.34.253" port: 40555 } } peers { permanent_uuid: "9cd1628c92d34b9a84fb83656c4961b4" member_type: VOTER last_known_addr { host: "127.12.34.252" port: 40371 } } }
I20260926 15:14:36.054322 2630 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a1918de22234eac8d9214ca9afff005 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 15:14:36.056505 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0e3f86c440934790aeb23b0ae70b3ebb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "5a1918de22234eac8d9214ca9afff005"
I20260926 15:14:36.056991 2480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5a1918de22234eac8d9214ca9afff005 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:14:36.060655 2480 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5a1918de22234eac8d9214ca9afff005 [term 1 FOLLOWER]: Refusing update from remote peer 0e3f86c440934790aeb23b0ae70b3ebb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:14:36.066633 2481 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5a1918de22234eac8d9214ca9afff005 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e3f86c440934790aeb23b0ae70b3ebb in term 1.
I20260926 15:14:36.068499 2506 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5a1918de22234eac8d9214ca9afff005 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:14:36.069041 2616 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0e3f86c440934790aeb23b0ae70b3ebb [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a1918de22234eac8d9214ca9afff005" member_type: VOTER last_known_addr { host: "127.12.34.253" port: 40555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:14:36.074398 2622 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 15:14:36.076201 2622 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:14:36.088920 2622 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0e3f86c440934790aeb23b0ae70b3ebb: Generated new TSK 0
I20260926 15:14:36.089947 2622 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 15:14:36.110018 2635 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a1918de22234eac8d9214ca9afff005 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0e3f86c440934790aeb23b0ae70b3ebb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e3f86c440934790aeb23b0ae70b3ebb" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 46301 } } peers { permanent_uuid: "5a1918de22234eac8d9214ca9afff005" member_type: VOTER last_known_addr { host: "127.12.34.253" port: 40555 } } peers { permanent_uuid: "9cd1628c92d34b9a84fb83656c4961b4" member_type: VOTER last_known_addr { host: "127.12.34.252" port: 40371 } } }
I20260926 15:14:36.110502 2635 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a1918de22234eac8d9214ca9afff005 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 15:14:36.110872 2630 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a1918de22234eac8d9214ca9afff005 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0e3f86c440934790aeb23b0ae70b3ebb. Latest consensus state: current_term: 1 leader_uuid: "0e3f86c440934790aeb23b0ae70b3ebb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e3f86c440934790aeb23b0ae70b3ebb" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 46301 } } peers { permanent_uuid: "5a1918de22234eac8d9214ca9afff005" member_type: VOTER last_known_addr { host: "127.12.34.253" port: 40555 } } peers { permanent_uuid: "9cd1628c92d34b9a84fb83656c4961b4" member_type: VOTER last_known_addr { host: "127.12.34.252" port: 40371 } } }
I20260926 15:14:36.111164 2630 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a1918de22234eac8d9214ca9afff005 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 15:14:36.112500 2616 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e3f86c440934790aeb23b0ae70b3ebb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0e3f86c440934790aeb23b0ae70b3ebb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e3f86c440934790aeb23b0ae70b3ebb" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 46301 } } peers { permanent_uuid: "5a1918de22234eac8d9214ca9afff005" member_type: VOTER last_known_addr { host: "127.12.34.253" port: 40555 } } peers { permanent_uuid: "9cd1628c92d34b9a84fb83656c4961b4" member_type: VOTER last_known_addr { host: "127.12.34.252" port: 40371 } } }
I20260926 15:14:36.112794 2616 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e3f86c440934790aeb23b0ae70b3ebb [sys.catalog]: This master's current role is: LEADER
I20260926 15:14:36.172544 2648 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20260926 15:14:36.196446 2583 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:14:36.187494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33122 (local address 127.12.34.252:40371)
0926 15:14:36.187670 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0926 15:14:36.187676 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:14:36.194684 (+ 7008us) server_negotiation.cc:407] Connection header received
0926 15:14:36.194900 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:14:36.194906 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:14:36.194992 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:14:36.195098 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:14:36.196175 (+ 1077us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33122: BlockingRecv error: recv got EOF from 127.0.0.1:33122 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260926 15:14:36.198267 2584 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:14:36.187060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53060 (local address 127.12.34.254:46301)
0926 15:14:36.187272 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0926 15:14:36.187281 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 15:14:36.196327 (+ 9046us) server_negotiation.cc:407] Connection header received
0926 15:14:36.196623 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:14:36.196627 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:14:36.196696 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:14:36.196783 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:14:36.198053 (+ 1270us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53060: BlockingRecv error: recv got EOF from 127.0.0.1:53060 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260926 15:14:36.198380 2510 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:14:36.195661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38534 (local address 127.12.34.253:40555)
0926 15:14:36.195878 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0926 15:14:36.195884 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:14:36.197110 (+ 1226us) server_negotiation.cc:407] Connection header received
0926 15:14:36.197219 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:14:36.197225 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:14:36.197309 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:14:36.197402 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:14:36.198178 (+ 776us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38534: BlockingRecv error: recv got EOF from 127.0.0.1:38534 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260926 15:14:36.211108 2510 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:14:36.206847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38536 (local address 127.12.34.253:40555)
0926 15:14:36.207003 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0926 15:14:36.207008 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:14:36.209609 (+ 2601us) server_negotiation.cc:407] Connection header received
0926 15:14:36.209868 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:14:36.209874 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:14:36.209986 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:14:36.210099 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:14:36.210851 (+ 752us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38536: BlockingRecv error: recv got EOF from 127.0.0.1:38536 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260926 15:14:36.223673 2510 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:14:36.218488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38548 (local address 127.12.34.253:40555)
0926 15:14:36.218641 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0926 15:14:36.218647 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:14:36.221676 (+ 3029us) server_negotiation.cc:407] Connection header received
0926 15:14:36.222231 (+ 555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:14:36.222237 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:14:36.222348 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:14:36.222450 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:14:36.223462 (+ 1012us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38548: BlockingRecv error: recv got EOF from 127.0.0.1:38548 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260926 15:14:36.284971 2510 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:14:36.270845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38564 (local address 127.12.34.253:40555)
0926 15:14:36.271114 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0926 15:14:36.271122 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 15:14:36.282301 (+ 11179us) server_negotiation.cc:407] Connection header received
0926 15:14:36.282924 (+ 623us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:14:36.282932 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:14:36.283073 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:14:36.283163 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:14:36.284632 (+ 1469us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38564: BlockingRecv error: recv got EOF from 127.0.0.1:38564 (error 108)
Metrics: {"server-negotiator.queue_time_us":96}
W20260926 15:14:36.335073 2510 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:14:36.321407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38578 (local address 127.12.34.253:40555)
0926 15:14:36.321678 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0926 15:14:36.321686 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 15:14:36.332628 (+ 10942us) server_negotiation.cc:407] Connection header received
0926 15:14:36.333272 (+ 644us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:14:36.333303 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:14:36.333449 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:14:36.333543 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:14:36.334777 (+ 1234us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38578: BlockingRecv error: recv got EOF from 127.0.0.1:38578 (error 108)
Metrics: {"server-negotiator.queue_time_us":114}
W20260926 15:14:36.411578 2510 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:14:36.401600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38588 (local address 127.12.34.253:40555)
0926 15:14:36.401830 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0926 15:14:36.401837 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:14:36.409786 (+ 7949us) server_negotiation.cc:407] Connection header received
0926 15:14:36.410044 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:14:36.410051 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:14:36.410160 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:14:36.410244 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:14:36.411327 (+ 1083us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38588: BlockingRecv error: recv got EOF from 127.0.0.1:38588 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260926 15:14:36.485419 2510 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:14:36.472246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38604 (local address 127.12.34.253:40555)
0926 15:14:36.472439 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0926 15:14:36.472446 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:14:36.483290 (+ 10844us) server_negotiation.cc:407] Connection header received
0926 15:14:36.483850 (+ 560us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:14:36.483857 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:14:36.483973 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:14:36.484063 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:14:36.485159 (+ 1096us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38604: BlockingRecv error: recv got EOF from 127.0.0.1:38604 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260926 15:14:36.565568 2510 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:14:36.551923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38610 (local address 127.12.34.253:40555)
0926 15:14:36.552105 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0926 15:14:36.552111 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:14:36.563339 (+ 11228us) server_negotiation.cc:407] Connection header received
0926 15:14:36.564000 (+ 661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:14:36.564006 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:14:36.564114 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:14:36.564197 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:14:36.565306 (+ 1109us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38610: BlockingRecv error: recv got EOF from 127.0.0.1:38610 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260926 15:14:36.644220 2510 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:14:36.642175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38612 (local address 127.12.34.253:40555)
0926 15:14:36.642469 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0926 15:14:36.642478 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 15:14:36.642749 (+ 271us) server_negotiation.cc:407] Connection header received
0926 15:14:36.642926 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:14:36.642935 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:14:36.643052 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:14:36.643182 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:14:36.643903 (+ 721us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38612: BlockingRecv error: recv got EOF from 127.0.0.1:38612 (error 108)
Metrics: {"server-negotiator.queue_time_us":98}
W20260926 15:14:36.777913 2510 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:14:36.775748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38620 (local address 127.12.34.253:40555)
0926 15:14:36.776031 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0926 15:14:36.776038 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:14:36.776333 (+ 295us) server_negotiation.cc:407] Connection header received
0926 15:14:36.776482 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:14:36.776486 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:14:36.776573 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:14:36.776682 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:14:36.777673 (+ 991us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38620: BlockingRecv error: recv got EOF from 127.0.0.1:38620 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
I20260926 15:14:36.870198 2598 catalog_manager.cc:1259] Loaded cluster ID: 276162dc3fbd4b83b92b7d74e5f94540
I20260926 15:14:36.870335 2598 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9cd1628c92d34b9a84fb83656c4961b4: loading cluster ID for follower catalog manager: success
I20260926 15:14:36.872776 2598 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9cd1628c92d34b9a84fb83656c4961b4: acquiring CA information for follower catalog manager: success
I20260926 15:14:36.874531 2598 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9cd1628c92d34b9a84fb83656c4961b4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260926 15:14:37.040302 2510 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:14:37.038447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38622 (local address 127.12.34.253:40555)
0926 15:14:37.038662 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0926 15:14:37.038669 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:14:37.038828 (+ 159us) server_negotiation.cc:407] Connection header received
0926 15:14:37.039057 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:14:37.039063 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:14:37.039145 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:14:37.039306 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:14:37.040078 (+ 772us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38622: BlockingRecv error: recv got EOF from 127.0.0.1:38622 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
I20260926 15:14:37.176743 2648 catalog_manager.cc:1259] Loaded cluster ID: 276162dc3fbd4b83b92b7d74e5f94540
I20260926 15:14:37.176859 2648 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5a1918de22234eac8d9214ca9afff005: loading cluster ID for follower catalog manager: success
I20260926 15:14:37.179602 2648 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5a1918de22234eac8d9214ca9afff005: acquiring CA information for follower catalog manager: success
I20260926 15:14:37.181490 2648 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5a1918de22234eac8d9214ca9afff005: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 15:14:37.564483 2710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:14:37.557219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38626 (local address 127.12.34.253:40555)
0926 15:14:37.557904 (+ 685us) server_negotiation.cc:206] Beginning negotiation
0926 15:14:37.557910 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:14:37.557941 (+ 31us) server_negotiation.cc:407] Connection header received
0926 15:14:37.558032 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:14:37.558037 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:14:37.558252 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:14:37.558418 (+ 166us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 15:14:37.559047 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:14:37.559924 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:14:37.561504 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:14:37.561997 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:14:37.562120 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:14:37.562984 (+ 864us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 15:14:37.563622 (+ 638us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 15:14:37.563753 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0926 15:14:37.564101 (+ 348us) server_negotiation.cc:299] Negotiation successful
0926 15:14:37.564187 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":528,"thread_start_us":384,"threads_started":1}
I20260926 15:14:37.578418 2716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:14:37.572425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33124 (local address 127.12.34.252:40371)
0926 15:14:37.573028 (+ 603us) server_negotiation.cc:206] Beginning negotiation
0926 15:14:37.573034 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:14:37.573055 (+ 21us) server_negotiation.cc:407] Connection header received
0926 15:14:37.573250 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:14:37.573254 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:14:37.573411 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:14:37.573611 (+ 200us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 15:14:37.574045 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:14:37.574784 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:14:37.576125 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:14:37.576526 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:14:37.576701 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:14:37.577244 (+ 543us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 15:14:37.577781 (+ 537us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 15:14:37.577950 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0926 15:14:37.578017 (+ 67us) server_negotiation.cc:299] Negotiation successful
0926 15:14:37.578100 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":475,"thread_start_us":313,"threads_started":1}
I20260926 15:14:37.593719 2716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:14:37.585929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33140 (local address 127.12.34.252:40371)
0926 15:14:37.586194 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0926 15:14:37.586203 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 15:14:37.587445 (+ 1242us) server_negotiation.cc:407] Connection header received
0926 15:14:37.587691 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:14:37.587697 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:14:37.587799 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:14:37.587864 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 15:14:37.588661 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:14:37.589782 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:14:37.591794 (+ 2012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:14:37.592446 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:14:37.592570 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:14:37.592940 (+ 370us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 15:14:37.593052 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 15:14:37.593153 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0926 15:14:37.593352 (+ 199us) server_negotiation.cc:299] Negotiation successful
0926 15:14:37.593436 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 15:14:37.593850 2710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:14:37.587581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38632 (local address 127.12.34.253:40555)
0926 15:14:37.587803 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0926 15:14:37.587811 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 15:14:37.588372 (+ 561us) server_negotiation.cc:407] Connection header received
0926 15:14:37.588587 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:14:37.588593 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:14:37.588676 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:14:37.588774 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 15:14:37.589308 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:14:37.590204 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:14:37.591772 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:14:37.592377 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:14:37.592502 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:14:37.593022 (+ 520us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 15:14:37.593216 (+ 194us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 15:14:37.593319 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 15:14:37.593515 (+ 196us) server_negotiation.cc:299] Negotiation successful
0926 15:14:37.593592 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260926 15:14:37.593845 2722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:14:37.585716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53066 (local address 127.12.34.254:46301)
0926 15:14:37.586956 (+ 1240us) server_negotiation.cc:206] Beginning negotiation
0926 15:14:37.586964 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 15:14:37.586984 (+ 20us) server_negotiation.cc:407] Connection header received
0926 15:14:37.587116 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:14:37.587124 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:14:37.587339 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:14:37.588025 (+ 686us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 15:14:37.588194 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:14:37.589364 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:14:37.591401 (+ 2037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:14:37.592021 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:14:37.592146 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:14:37.592658 (+ 512us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 15:14:37.593185 (+ 527us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 15:14:37.593309 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0926 15:14:37.593407 (+ 98us) server_negotiation.cc:299] Negotiation successful
0926 15:14:37.593484 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1111,"thread_start_us":992,"threads_started":1}
I20260926 15:14:37.599129 12427 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 2365
I20260926 15:14:37.600244 2427 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:14:37.709173 12427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8uoJGv/build/asan/bin/kudu with pid 2365
I20260926 15:14:37.729909 12427 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 2439
I20260926 15:14:37.731043 2501 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:14:37.836906 12427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8uoJGv/build/asan/bin/kudu with pid 2439
I20260926 15:14:37.857400 12427 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 2513
I20260926 15:14:37.858477 2575 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:14:37.982286 12427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8uoJGv/build/asan/bin/kudu with pid 2513
2026-09-26T15:14:37Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (5069 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790435318278681-12427-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 15:14:38 dist-test-slave-865v krb5kdc[2730](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 15:14:38 dist-test-slave-865v krb5kdc[2730](info): set up 2 sockets
Sep 26 15:14:38 dist-test-slave-865v krb5kdc[2730](info): commencing operation
krb5kdc: starting...
W20260926 15:14:39.105685 12427 mini_kdc.cc:121] Time spent starting KDC: real 1.081s user 0.005s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790435318278681-12427-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790435318278681-12427-0/krb5kdc/test-user.keytab.
Sep 26 15:14:39 dist-test-slave-865v krb5kdc[2730](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435679, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T15:14:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T15:14:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.34.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.34.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.34.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.254.keytab.
Entry for principal kudu/127.12.34.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.254.keytab.
I20260926 15:14:39.357324 12427 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8uoJGv/build/asan/bin/kudu
/tmp/dist-test-task8uoJGv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790435318278681-12427-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790435318278681-12427-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790435318278681-12427-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.34.254:36943
--webserver_interface=127.12.34.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.254.keytab
--principal=kudu/127.12.34.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.34.212:40715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.34.254:36943,127.12.34.253:42593,127.12.34.252:45443
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790435318278681-12427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790435318278681-12427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790435318278681-12427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:14:39.674624 2743 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 15:14:39.675024 2743 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:14:39.675158 2743 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:14:39.675305 2743 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:14:39.682674 2743 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:14:39.682830 2743 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:14:39.682942 2743 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:14:39.683036 2743 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:14:39.683127 2743 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:14:39.692945 2743 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.34.212:40715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790435318278681-12427-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790435318278681-12427-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.34.254:36943,127.12.34.253:42593,127.12.34.252:45443
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.254.keytab
--principal=kudu/127.12.34.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.34.254:36943
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.34.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790435318278681-12427-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:14:39.694271 2743 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:14:39.695564 2743 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:14:39.706974 2749 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:14:39.707070 2748 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:14:39.707019 2751 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:14:39.707823 2743 server_base.cc:1034] running on GCE node
I20260926 15:14:39.708591 2743 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:14:39.709947 2743 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:14:39.711191 2743 hybrid_clock.cc:648] HybridClock initialized: now 1790435679711123 us; error 46 us; skew 500 ppm
I20260926 15:14:39.711496 2743 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:14:39 dist-test-slave-865v krb5kdc[2730](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435679, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:14:39.720707 2743 init.cc:377] Logged in from keytab as kudu/127.12.34.254@KRBTEST.COM (short username kudu)
I20260926 15:14:39.722229 2743 webserver.cc:460] Webserver started at http://127.12.34.254:40507/ using document root <none> and password file <none>
I20260926 15:14:39.722898 2743 fs_manager.cc:359] Metadata directory not provided
I20260926 15:14:39.723027 2743 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:14:39.723341 2743 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:14:39.726239 2743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790435318278681-12427-0/minicluster-data/master-0/data/instance:
uuid: "8b1323a33a7b40309605b4a0215dc594"
format_stamp: "Formatted at 2026-09-26 15:14:39 on dist-test-slave-865v"
server_key: "ba552ff3082388fb18b924c3197d5354"
server_key_iv: "967a922bcb354c38106d652a1380e0b3"
server_key_version: "encryptionkey@0"
I20260926 15:14:39.726992 2743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790435318278681-12427-0/minicluster-data/master-0/wal/instance:
uuid: "8b1323a33a7b40309605b4a0215dc594"
format_stamp: "Formatted at 2026-09-26 15:14:39 on dist-test-slave-865v"
server_key: "ba552ff3082388fb18b924c3197d5354"
server_key_iv: "967a922bcb354c38106d652a1380e0b3"
server_key_version: "encryptionkey@0"
I20260926 15:14:39.731930 2743 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260926 15:14:39.735411 2758 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:14:39.736776 2743 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 15:14:39.736969 2743 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790435318278681-12427-0/minicluster-data/master-0/data,/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790435318278681-12427-0/minicluster-data/master-0/wal
uuid: "8b1323a33a7b40309605b4a0215dc594"
format_stamp: "Formatted at 2026-09-26 15:14:39 on dist-test-slave-865v"
server_key: "ba552ff3082388fb18b924c3197d5354"
server_key_iv: "967a922bcb354c38106d652a1380e0b3"
server_key_version: "encryptionkey@0"
I20260926 15:14:39.737250 2743 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790435318278681-12427-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790435318278681-12427-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790435318278681-12427-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:14:39.757129 2743 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:14:39.761298 2743 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:14:39.761576 2743 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:14:39.781173 2743 rpc_server.cc:307] RPC server started. Bound to: 127.12.34.254:36943
I20260926 15:14:39.781193 2809 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.34.254:36943 every 8 connection(s)
I20260926 15:14:39.782541 2743 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790435318278681-12427-0/minicluster-data/master-0/data/info.pb
I20260926 15:14:39.786170 2810 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:14:39.791394 12427 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8uoJGv/build/asan/bin/kudu as pid 2743
I20260926 15:14:39.791574 12427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790435318278681-12427-0/minicluster-data/master-0/wal/instance
I20260926 15:14:39.792094 12427 external_mini_cluster.cc:1428] Setting key 907f05d92209a2d132930ee93357797e
I20260926 15:14:39.794020 2810 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.34.254" port: 36943 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 15:14:39 dist-test-slave-865v krb5kdc[2730](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435679, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.254@KRBTEST.COM for kudu/127.12.34.254@KRBTEST.COM
I20260926 15:14:39.820322 2812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:14:39.797923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:36943 (local address 127.0.0.1:57258)
0926 15:14:39.798833 (+ 910us) negotiation.cc:107] Waiting for socket to connect
0926 15:14:39.798881 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 15:14:39.801676 (+ 2795us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:14:39.802769 (+ 1093us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:14:39.802773 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:14:39.803395 (+ 622us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:14:39.803876 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:14:39.803901 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:14:39.805509 (+ 1608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:14:39.805518 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:14:39.807393 (+ 1875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:14:39.807412 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:14:39.808334 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:14:39.808343 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:14:39.808554 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:14:39.809354 (+ 800us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:14:39.809388 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:14:39.811972 (+ 2584us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:14:39.816466 (+ 4494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:14:39.816475 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:14:39.816484 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:14:39.816899 (+ 415us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:14:39.817405 (+ 506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:14:39.817408 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:14:39.817410 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:14:39.817598 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:14:39.818592 (+ 994us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:14:39.818598 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:14:39.818749 (+ 151us) client_negotiation.cc:769] Sending connection context
0926 15:14:39.818949 (+ 200us) client_negotiation.cc:240] Negotiation successful
0926 15:14:39.819305 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":608,"thread_start_us":371,"threads_started":1}
WARNING: no policy specified for kudu/127.12.34.253@KRBTEST.COM; defaulting to no policy
I20260926 15:14:39.820353 2813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:14:39.797923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57258 (local address 127.12.34.254:36943)
0926 15:14:39.799767 (+ 1844us) server_negotiation.cc:206] Beginning negotiation
0926 15:14:39.799776 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 15:14:39.799816 (+ 40us) server_negotiation.cc:407] Connection header received
0926 15:14:39.802135 (+ 2319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:14:39.802144 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:14:39.802471 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:14:39.802723 (+ 252us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:14:39.804094 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:14:39.805351 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:14:39.807546 (+ 2195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:14:39.808088 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:14:39.808219 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:14:39.812210 (+ 3991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:14:39.812222 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:14:39.812240 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:14:39.812281 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:14:39.816323 (+ 4042us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:14:39.817041 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:14:39.817052 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:14:39.817058 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:14:39.817283 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:14:39.817718 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:14:39.817725 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:14:39.817731 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:14:39.818422 (+ 691us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:14:39.818623 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0926 15:14:39.819135 (+ 512us) server_negotiation.cc:299] Negotiation successful
0926 15:14:39.819504 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":64,"server-negotiator.queue_time_us":1666,"spinlock_wait_cycles":36736,"thread_start_us":1428,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.34.253@KRBTEST.COM" created.
I20260926 15:14:39.823606 2810 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.34.253" port: 42593 } has no permanent_uuid. Determining permanent_uuid...
I20260926 15:14:39.824890 2812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:14:39.824492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.253:42593 (local address 127.0.0.1:41814)
0926 15:14:39.824660 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0926 15:14:39.824780 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.253:42593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260926 15:14:39.825245 2762 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.34.253:42593: connect: Connection refused (error 111)
I20260926 15:14:39.826848 2812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:14:39.826437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.253:42593 (local address 127.0.0.1:41820)
0926 15:14:39.826666 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0926 15:14:39.826732 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.253:42593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140}
W20260926 15:14:39.827097 2810 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.34.253:42593: Network error: Client connection negotiation failed: client connection to 127.12.34.253:42593: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.34.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.253.keytab.
Entry for principal kudu/127.12.34.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.253.keytab.
I20260926 15:14:39.857695 12427 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8uoJGv/build/asan/bin/kudu
/tmp/dist-test-task8uoJGv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790435318278681-12427-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790435318278681-12427-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790435318278681-12427-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790435318278681-12427-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.34.253:42593
--webserver_interface=127.12.34.253
--webserver_port=0
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.253.keytab
--principal=kudu/127.12.34.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.34.212:40715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.34.254:36943,127.12.34.253:42593,127.12.34.252:45443
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790435318278681-12427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790435318278681-12427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790435318278681-12427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 15:14:39.876454 2810 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.34.253" port: 42593 } attempt: 1
I20260926 15:14:39.877720 2812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:14:39.877261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.253:42593 (local address 127.0.0.1:41824)
0926 15:14:39.877463 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0926 15:14:39.877584 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.253:42593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260926 15:14:39.878759 2812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:14:39.878428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.253:42593 (local address 127.0.0.1:41836)
0926 15:14:39.878569 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0926 15:14:39.878636 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.253:42593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260926 15:14:39.879077 2810 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.34.253:42593: Network error: Client connection negotiation failed: client connection to 127.12.34.253:42593: connect: Connection refused (error 111)
I20260926 15:14:39.947381 2810 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.34.253" port: 42593 } attempt: 2
I20260926 15:14:39.948633 2812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:14:39.948107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.253:42593 (local address 127.0.0.1:41838)
0926 15:14:39.948376 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0926 15:14:39.948472 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.253:42593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
I20260926 15:14:39.949890 2812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:14:39.949465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.253:42593 (local address 127.0.0.1:41840)
0926 15:14:39.949640 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0926 15:14:39.949737 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.253:42593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260926 15:14:39.950134 2810 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.34.253:42593: Network error: Client connection negotiation failed: client connection to 127.12.34.253:42593: connect: Connection refused (error 111)
I20260926 15:14:40.041424 2810 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.34.253" port: 42593 } attempt: 3
I20260926 15:14:40.042776 2812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:14:40.042172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.253:42593 (local address 127.0.0.1:41844)
0926 15:14:40.042432 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0926 15:14:40.042580 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.253:42593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260926 15:14:40.043953 2812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:14:40.043458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.253:42593 (local address 127.0.0.1:41854)
0926 15:14:40.043661 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0926 15:14:40.043765 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.253:42593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260926 15:14:40.044216 2810 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.34.253:42593: Network error: Client connection negotiation failed: client connection to 127.12.34.253:42593: connect: Connection refused (error 111)
I20260926 15:14:40.187505 2810 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.34.253" port: 42593 } attempt: 4
I20260926 15:14:40.188762 2812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:14:40.188178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.253:42593 (local address 127.0.0.1:41862)
0926 15:14:40.188505 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0926 15:14:40.188605 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.253:42593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167}
I20260926 15:14:40.189838 2812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:14:40.189459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.253:42593 (local address 127.0.0.1:41868)
0926 15:14:40.189609 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0926 15:14:40.189703 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.253:42593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260926 15:14:40.190054 2810 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.34.253:42593: Network error: Client connection negotiation failed: client connection to 127.12.34.253:42593: connect: Connection refused (error 111)
W20260926 15:14:40.191478 2817 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 15:14:40.191766 2817 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:14:40.191874 2817 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:14:40.191998 2817 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:14:40.200057 2817 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:14:40.200147 2817 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:14:40.200205 2817 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:14:40.200248 2817 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:14:40.200304 2817 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:14:40.210423 2817 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.34.212:40715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790435318278681-12427-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790435318278681-12427-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.12.34.254:36943,127.12.34.253:42593,127.12.34.252:45443
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.253.keytab
--principal=kudu/127.12.34.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.34.253:42593
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790435318278681-12427-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.12.34.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790435318278681-12427-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:14:40.212208 2817 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:14:40.213567 2817 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:14:40.225863 2822 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:14:40.226017 2823 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:14:40.226012 2825 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:14:40.227198 2817 server_base.cc:1034] running on GCE node
I20260926 15:14:40.228003 2817 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:14:40.229467 2817 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:14:40.230844 2817 hybrid_clock.cc:648] HybridClock initialized: now 1790435680230718 us; error 34 us; skew 500 ppm
I20260926 15:14:40.231451 2817 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:14:40 dist-test-slave-865v krb5kdc[2730](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435680, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:14:40.235101 2817 init.cc:377] Logged in from keytab as kudu/127.12.34.253@KRBTEST.COM (short username kudu)
I20260926 15:14:40.236729 2817 webserver.cc:460] Webserver started at http://127.12.34.253:34445/ using document root <none> and password file <none>
I20260926 15:14:40.237468 2817 fs_manager.cc:359] Metadata directory not provided
I20260926 15:14:40.237557 2817 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:14:40.237820 2817 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:14:40.240783 2817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790435318278681-12427-0/minicluster-data/master-1/data/instance:
uuid: "edbd7d56c12245ef81d230093e189a24"
format_stamp: "Formatted at 2026-09-26 15:14:40 on dist-test-slave-865v"
server_key: "eef3136771d8619a56f66b750acbe856"
server_key_iv: "4288803c4a759074b471f508d6b7566d"
server_key_version: "encryptionkey@0"
I20260926 15:14:40.241626 2817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790435318278681-12427-0/minicluster-data/master-1/wal/instance:
uuid: "edbd7d56c12245ef81d230093e189a24"
format_stamp: "Formatted at 2026-09-26 15:14:40 on dist-test-slave-865v"
server_key: "eef3136771d8619a56f66b750acbe856"
server_key_iv: "4288803c4a759074b471f508d6b7566d"
server_key_version: "encryptionkey@0"
I20260926 15:14:40.246699 2817 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 15:14:40.250097 2832 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:14:40.251807 2817 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 15:14:40.252243 2817 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790435318278681-12427-0/minicluster-data/master-1/data,/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790435318278681-12427-0/minicluster-data/master-1/wal
uuid: "edbd7d56c12245ef81d230093e189a24"
format_stamp: "Formatted at 2026-09-26 15:14:40 on dist-test-slave-865v"
server_key: "eef3136771d8619a56f66b750acbe856"
server_key_iv: "4288803c4a759074b471f508d6b7566d"
server_key_version: "encryptionkey@0"
I20260926 15:14:40.252669 2817 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790435318278681-12427-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790435318278681-12427-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790435318278681-12427-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:14:40.274492 2817 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:14:40.278913 2817 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:14:40.279207 2817 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:14:40.298990 2817 rpc_server.cc:307] RPC server started. Bound to: 127.12.34.253:42593
I20260926 15:14:40.299011 2883 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.34.253:42593 every 8 connection(s)
I20260926 15:14:40.300325 2817 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790435318278681-12427-0/minicluster-data/master-1/data/info.pb
I20260926 15:14:40.304376 2884 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:14:40.304491 12427 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8uoJGv/build/asan/bin/kudu as pid 2817
I20260926 15:14:40.304675 12427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790435318278681-12427-0/minicluster-data/master-1/wal/instance
I20260926 15:14:40.305262 12427 external_mini_cluster.cc:1428] Setting key c4d9394d5bf24bb07cdc415f20e1c27c
I20260926 15:14:40.314656 2884 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.34.254" port: 36943 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 15:14:40 dist-test-slave-865v krb5kdc[2730](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435680, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.253@KRBTEST.COM for kudu/127.12.34.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.12.34.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.34.252@KRBTEST.COM" created.
I20260926 15:14:40.339398 2813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:14:40.318268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57274 (local address 127.12.34.254:36943)
0926 15:14:40.318498 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0926 15:14:40.318504 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:14:40.319801 (+ 1297us) server_negotiation.cc:407] Connection header received
0926 15:14:40.321966 (+ 2165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:14:40.321976 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:14:40.322084 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:14:40.322225 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:14:40.323925 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:14:40.325359 (+ 1434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:14:40.327566 (+ 2207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:14:40.328362 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:14:40.328523 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:14:40.332620 (+ 4097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:14:40.332625 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:14:40.332628 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:14:40.332683 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:14:40.335979 (+ 3296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:14:40.336949 (+ 970us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:14:40.336953 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:14:40.336956 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:14:40.337072 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:14:40.337649 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:14:40.337652 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:14:40.337654 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:14:40.337946 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:14:40.338057 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0926 15:14:40.339004 (+ 947us) server_negotiation.cc:299] Negotiation successful
0926 15:14:40.339167 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260926 15:14:40.341572 2887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:14:40.318791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:36943 (local address 127.0.0.1:57274)
0926 15:14:40.319620 (+ 829us) negotiation.cc:107] Waiting for socket to connect
0926 15:14:40.319663 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 15:14:40.321507 (+ 1844us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:14:40.322466 (+ 959us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:14:40.322475 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:14:40.323072 (+ 597us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:14:40.323624 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:14:40.323640 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:14:40.325688 (+ 2048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:14:40.325695 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:14:40.327307 (+ 1612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:14:40.327332 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:14:40.328626 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:14:40.328634 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:14:40.328905 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:14:40.329817 (+ 912us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:14:40.329842 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:14:40.332364 (+ 2522us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:14:40.336130 (+ 3766us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:14:40.336141 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:14:40.336160 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:14:40.336816 (+ 656us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:14:40.337274 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:14:40.337281 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:14:40.337287 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:14:40.337528 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:14:40.338099 (+ 571us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:14:40.338110 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:14:40.338585 (+ 475us) client_negotiation.cc:769] Sending connection context
0926 15:14:40.338890 (+ 305us) client_negotiation.cc:240] Negotiation successful
0926 15:14:40.340089 (+ 1199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"thread_start_us":375,"threads_started":1}
I20260926 15:14:40.343361 2884 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.34.253" port: 42593 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 15:14:40 dist-test-slave-865v krb5kdc[2730](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435680, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.253@KRBTEST.COM for kudu/127.12.34.253@KRBTEST.COM
I20260926 15:14:40.360383 2887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:14:40.344133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.253:42593 (local address 127.0.0.1:41878)
0926 15:14:40.344326 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0926 15:14:40.344348 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 15:14:40.344456 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:14:40.345501 (+ 1045us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:14:40.345512 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:14:40.345997 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:14:40.346129 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:14:40.346137 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:14:40.348283 (+ 2146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:14:40.348287 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:14:40.349192 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:14:40.349203 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:14:40.350422 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:14:40.350426 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:14:40.350560 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:14:40.351235 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:14:40.351257 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:14:40.353569 (+ 2312us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:14:40.357794 (+ 4225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:14:40.357799 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:14:40.357802 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:14:40.358277 (+ 475us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:14:40.358712 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:14:40.358715 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:14:40.358717 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:14:40.358853 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:14:40.359586 (+ 733us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:14:40.359590 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:14:40.359716 (+ 126us) client_negotiation.cc:769] Sending connection context
0926 15:14:40.359831 (+ 115us) client_negotiation.cc:240] Negotiation successful
0926 15:14:40.360126 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260926 15:14:40.360818 2888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:14:40.344351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41878 (local address 127.12.34.253:42593)
0926 15:14:40.345041 (+ 690us) server_negotiation.cc:206] Beginning negotiation
0926 15:14:40.345047 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:14:40.345071 (+ 24us) server_negotiation.cc:407] Connection header received
0926 15:14:40.345166 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:14:40.345172 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:14:40.345341 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:14:40.345459 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:14:40.346310 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:14:40.348140 (+ 1830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:14:40.349325 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:14:40.350272 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:14:40.350410 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:14:40.353803 (+ 3393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:14:40.353819 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:14:40.353838 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:14:40.353878 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:14:40.357639 (+ 3761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:14:40.358426 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:14:40.358437 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:14:40.358444 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:14:40.358592 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:14:40.358970 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:14:40.358977 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:14:40.358983 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:14:40.359452 (+ 469us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:14:40.359665 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0926 15:14:40.360034 (+ 369us) server_negotiation.cc:299] Negotiation successful
0926 15:14:40.360333 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":542,"thread_start_us":381,"threads_started":1}
I20260926 15:14:40.363124 2884 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.34.252" port: 45443 } has no permanent_uuid. Determining permanent_uuid...
I20260926 15:14:40.364346 2887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:14:40.363917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.252:45443 (local address 127.0.0.1:45250)
0926 15:14:40.364105 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0926 15:14:40.364212 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.252:45443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260926 15:14:40.364657 2834 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.34.252:45443: connect: Connection refused (error 111)
I20260926 15:14:40.366212 2887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:14:40.365896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.252:45443 (local address 127.0.0.1:45262)
0926 15:14:40.366038 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0926 15:14:40.366116 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.252:45443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260926 15:14:40.366456 2884 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.34.252:45443: Network error: Client connection negotiation failed: client connection to 127.12.34.252:45443: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.34.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.252.keytab.
Entry for principal kudu/127.12.34.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.252.keytab.
I20260926 15:14:40.373173 12427 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8uoJGv/build/asan/bin/kudu
/tmp/dist-test-task8uoJGv/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790435318278681-12427-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790435318278681-12427-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790435318278681-12427-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790435318278681-12427-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.34.252:45443
--webserver_interface=127.12.34.252
--webserver_port=0
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.252.keytab
--principal=kudu/127.12.34.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.34.212:40715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.34.254:36943,127.12.34.253:42593,127.12.34.252:45443
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790435318278681-12427-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790435318278681-12427-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790435318278681-12427-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 15:14:40.415779 2884 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.34.252" port: 45443 } attempt: 1
I20260926 15:14:40.417035 2887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:14:40.416540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.252:45443 (local address 127.0.0.1:45278)
0926 15:14:40.416777 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0926 15:14:40.416883 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.252:45443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260926 15:14:40.418123 2887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:14:40.417811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.252:45443 (local address 127.0.0.1:45280)
0926 15:14:40.417956 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0926 15:14:40.418027 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.252:45443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260926 15:14:40.418399 2884 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.34.252:45443: Network error: Client connection negotiation failed: client connection to 127.12.34.252:45443: connect: Connection refused (error 111)
I20260926 15:14:40.486655 2884 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.34.252" port: 45443 } attempt: 2
I20260926 15:14:40.487797 2887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:14:40.487342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.252:45443 (local address 127.0.0.1:45282)
0926 15:14:40.487551 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0926 15:14:40.487646 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.252:45443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260926 15:14:40.488768 2887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:14:40.488449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.252:45443 (local address 127.0.0.1:45288)
0926 15:14:40.488587 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0926 15:14:40.488662 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.252:45443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260926 15:14:40.489028 2884 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.34.252:45443: Network error: Client connection negotiation failed: client connection to 127.12.34.252:45443: connect: Connection refused (error 111)
I20260926 15:14:40.489301 2810 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.34.253" port: 42593 } attempt: 5
Sep 26 15:14:40 dist-test-slave-865v krb5kdc[2730](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435679, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.254@KRBTEST.COM for kudu/127.12.34.253@KRBTEST.COM
I20260926 15:14:40.504207 2812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:14:40.489882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.253:42593 (local address 127.0.0.1:41892)
0926 15:14:40.490091 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0926 15:14:40.490107 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 15:14:40.490219 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:14:40.490589 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:14:40.490592 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:14:40.490935 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:14:40.491141 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:14:40.491149 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:14:40.492691 (+ 1542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:14:40.492696 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:14:40.493637 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:14:40.493651 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:14:40.494750 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:14:40.494756 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:14:40.494955 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:14:40.495610 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:14:40.495628 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:14:40.497630 (+ 2002us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:14:40.501555 (+ 3925us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:14:40.501559 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:14:40.501562 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:14:40.501915 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:14:40.502339 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:14:40.502345 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:14:40.502348 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:14:40.502652 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:14:40.503474 (+ 822us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:14:40.503478 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:14:40.503629 (+ 151us) client_negotiation.cc:769] Sending connection context
0926 15:14:40.503749 (+ 120us) client_negotiation.cc:240] Negotiation successful
0926 15:14:40.503954 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260926 15:14:40.504493 2888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:14:40.489938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41892 (local address 127.12.34.253:42593)
0926 15:14:40.490111 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0926 15:14:40.490117 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:14:40.490164 (+ 47us) server_negotiation.cc:407] Connection header received
0926 15:14:40.490339 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:14:40.490344 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:14:40.490441 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:14:40.490562 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:14:40.491299 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:14:40.492514 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:14:40.493807 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:14:40.494499 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:14:40.494642 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:14:40.497845 (+ 3203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:14:40.497852 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:14:40.497857 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:14:40.497893 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:14:40.501374 (+ 3481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:14:40.502037 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:14:40.502042 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:14:40.502046 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:14:40.502185 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:14:40.502798 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:14:40.502806 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:14:40.502811 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:14:40.503250 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:14:40.503423 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0926 15:14:40.503913 (+ 490us) server_negotiation.cc:299] Negotiation successful
0926 15:14:40.504181 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260926 15:14:40.505551 2810 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.34.252" port: 45443 } has no permanent_uuid. Determining permanent_uuid...
I20260926 15:14:40.506671 2812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:14:40.506277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.252:45443 (local address 127.0.0.1:45304)
0926 15:14:40.506448 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0926 15:14:40.506543 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.252:45443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260926 15:14:40.508116 2812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:14:40.507381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.252:45443 (local address 127.0.0.1:45316)
0926 15:14:40.507722 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0926 15:14:40.507942 (+ 220us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.252:45443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
W20260926 15:14:40.508555 2810 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.34.252:45443: Network error: Client connection negotiation failed: client connection to 127.12.34.252:45443: connect: Connection refused (error 111)
I20260926 15:14:40.559888 2810 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.34.252" port: 45443 } attempt: 1
I20260926 15:14:40.561403 2812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:14:40.560716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.252:45443 (local address 127.0.0.1:45330)
0926 15:14:40.560947 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0926 15:14:40.561042 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.252:45443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260926 15:14:40.563123 2812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:14:40.562478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.252:45443 (local address 127.0.0.1:45346)
0926 15:14:40.562711 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0926 15:14:40.563013 (+ 302us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.252:45443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
W20260926 15:14:40.563459 2810 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.34.252:45443: Network error: Client connection negotiation failed: client connection to 127.12.34.252:45443: connect: Connection refused (error 111)
I20260926 15:14:40.580322 2884 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.34.252" port: 45443 } attempt: 3
I20260926 15:14:40.581763 2887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:14:40.581246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.252:45443 (local address 127.0.0.1:45350)
0926 15:14:40.581476 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0926 15:14:40.581578 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.252:45443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260926 15:14:40.582871 2887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:14:40.582421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.252:45443 (local address 127.0.0.1:45364)
0926 15:14:40.582663 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0926 15:14:40.582754 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.252:45443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137}
W20260926 15:14:40.583130 2884 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.34.252:45443: Network error: Client connection negotiation failed: client connection to 127.12.34.252:45443: connect: Connection refused (error 111)
I20260926 15:14:40.631703 2810 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.34.252" port: 45443 } attempt: 2
I20260926 15:14:40.632848 2812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:14:40.632364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.252:45443 (local address 127.0.0.1:45380)
0926 15:14:40.632604 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0926 15:14:40.632691 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.252:45443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260926 15:14:40.633869 2812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:14:40.633569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.252:45443 (local address 127.0.0.1:45390)
0926 15:14:40.633715 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0926 15:14:40.633782 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.252:45443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260926 15:14:40.634073 2810 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.34.252:45443: Network error: Client connection negotiation failed: client connection to 127.12.34.252:45443: connect: Connection refused (error 111)
W20260926 15:14:40.706753 2891 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 15:14:40.707120 2891 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:14:40.707252 2891 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:14:40.707392 2891 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:14:40.714537 2891 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:14:40.714675 2891 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:14:40.714775 2891 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:14:40.714875 2891 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:14:40.714962 2891 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:14:40.724586 2891 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.34.212:40715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790435318278681-12427-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790435318278681-12427-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.12.34.254:36943,127.12.34.253:42593,127.12.34.252:45443
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790435318278681-12427-0/krb5kdc/kudu_127.12.34.252.keytab
--principal=kudu/127.12.34.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.34.252:45443
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790435318278681-12427-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.12.34.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790435318278681-12427-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 14:12:06 UTC on ccycloud.quasar-ygbmzu.root.comops.site
build id 4674337
ASAN enabled
I20260926 15:14:40.725908 2891 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:14:40.726339 2884 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.34.252" port: 45443 } attempt: 4
I20260926 15:14:40.727478 2891 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 15:14:40.727661 2887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:14:40.727090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.252:45443 (local address 127.0.0.1:45400)
0926 15:14:40.727362 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0926 15:14:40.727469 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.252:45443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260926 15:14:40.728699 2887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:14:40.728341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.252:45443 (local address 127.0.0.1:45412)
0926 15:14:40.728495 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 15:14:40.728578 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.252:45443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260926 15:14:40.728946 2884 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.34.252:45443: Network error: Client connection negotiation failed: client connection to 127.12.34.252:45443: connect: Connection refused (error 111)
I20260926 15:14:40.740249 2810 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.34.252" port: 45443 } attempt: 3
W20260926 15:14:40.740692 2899 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:14:40.740715 2897 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:14:40.741755 2812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:14:40.741033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.252:45443 (local address 127.0.0.1:45422)
0926 15:14:40.741370 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0926 15:14:40.741500 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.252:45443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
W20260926 15:14:40.742339 2896 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:14:40.742962 2812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:14:40.742395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.252:45443 (local address 127.0.0.1:45434)
0926 15:14:40.742655 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0926 15:14:40.742778 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.34.252:45443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
W20260926 15:14:40.743263 2810 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.34.252:45443: Network error: Client connection negotiation failed: client connection to 127.12.34.252:45443: connect: Connection refused (error 111)
I20260926 15:14:40.746897 2891 server_base.cc:1034] running on GCE node
I20260926 15:14:40.747818 2891 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:14:40.749518 2891 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:14:40.750794 2891 hybrid_clock.cc:648] HybridClock initialized: now 1790435680750737 us; error 25 us; skew 500 ppm
I20260926 15:14:40.751164 2891 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 15:14:40 dist-test-slave-865v krb5kdc[2730](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435680, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:14:40.754598 2891 init.cc:377] Logged in from keytab as kudu/127.12.34.252@KRBTEST.COM (short username kudu)
I20260926 15:14:40.756306 2891 webserver.cc:460] Webserver started at http://127.12.34.252:45651/ using document root <none> and password file <none>
I20260926 15:14:40.757339 2891 fs_manager.cc:359] Metadata directory not provided
I20260926 15:14:40.757473 2891 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:14:40.757831 2891 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:14:40.760970 2891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790435318278681-12427-0/minicluster-data/master-2/data/instance:
uuid: "7b2aee94e90c4de98e7319e5d9d7b3e7"
format_stamp: "Formatted at 2026-09-26 15:14:40 on dist-test-slave-865v"
server_key: "9f1adb09f253d6ba94dd5c7296194844"
server_key_iv: "ae26ecf097087aa30486a27155251cd2"
server_key_version: "encryptionkey@0"
I20260926 15:14:40.761919 2891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790435318278681-12427-0/minicluster-data/master-2/wal/instance:
uuid: "7b2aee94e90c4de98e7319e5d9d7b3e7"
format_stamp: "Formatted at 2026-09-26 15:14:40 on dist-test-slave-865v"
server_key: "9f1adb09f253d6ba94dd5c7296194844"
server_key_iv: "ae26ecf097087aa30486a27155251cd2"
server_key_version: "encryptionkey@0"
I20260926 15:14:40.767433 2891 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 15:14:40.771492 2906 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:14:40.773238 2891 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 15:14:40.773443 2891 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790435318278681-12427-0/minicluster-data/master-2/data,/tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790435318278681-12427-0/minicluster-data/master-2/wal
uuid: "7b2aee94e90c4de98e7319e5d9d7b3e7"
format_stamp: "Formatted at 2026-09-26 15:14:40 on dist-test-slave-865v"
server_key: "9f1adb09f253d6ba94dd5c7296194844"
server_key_iv: "ae26ecf097087aa30486a27155251cd2"
server_key_version: "encryptionkey@0"
I20260926 15:14:40.773681 2891 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790435318278681-12427-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790435318278681-12427-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790435318278681-12427-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:14:40.798919 2891 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:14:40.803745 2891 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:14:40.804155 2891 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:14:40.824793 2891 rpc_server.cc:307] RPC server started. Bound to: 127.12.34.252:45443
I20260926 15:14:40.824797 2957 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.34.252:45443 every 8 connection(s)
I20260926 15:14:40.826297 2891 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790435318278681-12427-0/minicluster-data/master-2/data/info.pb
I20260926 15:14:40.828467 12427 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8uoJGv/build/asan/bin/kudu as pid 2891
I20260926 15:14:40.828688 12427 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8uoJGv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790435318278681-12427-0/minicluster-data/master-2/wal/instance
I20260926 15:14:40.829239 12427 external_mini_cluster.cc:1428] Setting key b530f123d879fc90bef77658bc33626e
I20260926 15:14:40.830325 2958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:14:40.840801 2958 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.34.254" port: 36943 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 15:14:40 dist-test-slave-865v krb5kdc[2730](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435679, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.34.252@KRBTEST.COM
Sep 26 15:14:40 dist-test-slave-865v krb5kdc[2730](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435680, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.252@KRBTEST.COM for kudu/127.12.34.254@KRBTEST.COM
I20260926 15:14:40.854537 2961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:14:40.830977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45450 (local address 127.12.34.252:45443)
0926 15:14:40.831804 (+ 827us) server_negotiation.cc:206] Beginning negotiation
0926 15:14:40.831815 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 15:14:40.831856 (+ 41us) server_negotiation.cc:407] Connection header received
0926 15:14:40.834197 (+ 2341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:14:40.834219 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:14:40.834658 (+ 439us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:14:40.834991 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:14:40.836748 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:14:40.838378 (+ 1630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:14:40.840122 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:14:40.840970 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:14:40.841189 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:14:40.846008 (+ 4819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:14:40.846015 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:14:40.846024 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:14:40.846064 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:14:40.849881 (+ 3817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:14:40.850751 (+ 870us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:14:40.850758 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:14:40.850764 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:14:40.851017 (+ 253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:14:40.851437 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:14:40.851449 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:14:40.851452 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:14:40.852225 (+ 773us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:14:40.852557 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0926 15:14:40.852897 (+ 340us) server_negotiation.cc:299] Negotiation successful
0926 15:14:40.853398 (+ 501us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":614,"thread_start_us":399,"threads_started":1}
I20260926 15:14:40.861374 2963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:14:40.843782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57288 (local address 127.12.34.254:36943)
0926 15:14:40.845223 (+ 1441us) server_negotiation.cc:206] Beginning negotiation
0926 15:14:40.845228 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:14:40.845243 (+ 15us) server_negotiation.cc:407] Connection header received
0926 15:14:40.845291 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:14:40.845294 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:14:40.845351 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:14:40.845438 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:14:40.846761 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:14:40.847518 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:14:40.850051 (+ 2533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:14:40.850770 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:14:40.850867 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:14:40.854744 (+ 3877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:14:40.854751 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:14:40.854755 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:14:40.854792 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:14:40.858194 (+ 3402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:14:40.859159 (+ 965us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:14:40.859166 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:14:40.859172 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:14:40.859341 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:14:40.859790 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:14:40.859795 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:14:40.859799 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:14:40.860126 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:14:40.860271 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0926 15:14:40.860836 (+ 565us) server_negotiation.cc:299] Negotiation successful
0926 15:14:40.860982 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1271,"thread_start_us":1156,"threads_started":1}
I20260926 15:14:40.861536 2962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:14:40.843709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.254:36943 (local address 127.0.0.1:57288)
0926 15:14:40.844473 (+ 764us) negotiation.cc:107] Waiting for socket to connect
0926 15:14:40.844516 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 15:14:40.844685 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:14:40.845543 (+ 858us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:14:40.845550 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:14:40.846278 (+ 728us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:14:40.846599 (+ 321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:14:40.846613 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:14:40.847757 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:14:40.847766 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:14:40.849821 (+ 2055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:14:40.849838 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:14:40.850964 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:14:40.850971 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:14:40.851232 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:14:40.851845 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:14:40.851868 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:14:40.854542 (+ 2674us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:14:40.858357 (+ 3815us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:14:40.858369 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:14:40.858375 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:14:40.859017 (+ 642us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:14:40.859491 (+ 474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:14:40.859495 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:14:40.859497 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:14:40.859661 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:14:40.860280 (+ 619us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:14:40.860290 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:14:40.860511 (+ 221us) client_negotiation.cc:769] Sending connection context
0926 15:14:40.860749 (+ 238us) client_negotiation.cc:240] Negotiation successful
0926 15:14:40.861062 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":318,"threads_started":1}
I20260926 15:14:40.863183 2958 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.34.253" port: 42593 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 15:14:40 dist-test-slave-865v krb5kdc[2730](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435679, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.34.254@KRBTEST.COM
Sep 26 15:14:40 dist-test-slave-865v krb5kdc[2730](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435680, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.252@KRBTEST.COM for kudu/127.12.34.253@KRBTEST.COM
I20260926 15:14:40.875932 2964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:14:40.857767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57296 (local address 127.12.34.254:36943)
0926 15:14:40.858493 (+ 726us) server_negotiation.cc:206] Beginning negotiation
0926 15:14:40.858502 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 15:14:40.858529 (+ 27us) server_negotiation.cc:407] Connection header received
0926 15:14:40.858595 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:14:40.858604 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:14:40.858680 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:14:40.858805 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:14:40.860318 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:14:40.861739 (+ 1421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:14:40.863361 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:14:40.864126 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:14:40.864327 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:14:40.870088 (+ 5761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:14:40.870098 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:14:40.870105 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:14:40.870161 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:14:40.873410 (+ 3249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:14:40.874038 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:14:40.874044 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:14:40.874047 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:14:40.874175 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:14:40.874527 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:14:40.874532 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:14:40.874536 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:14:40.874791 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:14:40.875092 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0926 15:14:40.875359 (+ 267us) server_negotiation.cc:299] Negotiation successful
0926 15:14:40.875570 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":546,"thread_start_us":259,"threads_started":1}
I20260926 15:14:40.879664 2962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:14:40.864149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.253:42593 (local address 127.0.0.1:41904)
0926 15:14:40.864706 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0926 15:14:40.864721 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 15:14:40.864806 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:14:40.865429 (+ 623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:14:40.865437 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:14:40.865837 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:14:40.865976 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:14:40.865983 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:14:40.867664 (+ 1681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:14:40.867668 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:14:40.868523 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:14:40.868537 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:14:40.869453 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:14:40.869460 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:14:40.869673 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:14:40.870286 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:14:40.870302 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:14:40.872570 (+ 2268us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:14:40.876631 (+ 4061us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:14:40.876636 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:14:40.876639 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:14:40.877083 (+ 444us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:14:40.877616 (+ 533us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:14:40.877624 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:14:40.877632 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:14:40.877860 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:14:40.878612 (+ 752us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:14:40.878617 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:14:40.878790 (+ 173us) client_negotiation.cc:769] Sending connection context
0926 15:14:40.878952 (+ 162us) client_negotiation.cc:240] Negotiation successful
0926 15:14:40.879256 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470}
I20260926 15:14:40.879693 2888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:14:40.864532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41904 (local address 127.12.34.253:42593)
0926 15:14:40.864758 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0926 15:14:40.864764 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:14:40.864784 (+ 20us) server_negotiation.cc:407] Connection header received
0926 15:14:40.864937 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:14:40.864944 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:14:40.865035 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:14:40.865187 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:14:40.866191 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:14:40.867491 (+ 1300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:14:40.868685 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:14:40.869287 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:14:40.869381 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:14:40.872787 (+ 3406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:14:40.872793 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:14:40.872796 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:14:40.872831 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:14:40.876485 (+ 3654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:14:40.877245 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:14:40.877250 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:14:40.877252 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:14:40.877442 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:14:40.878029 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:14:40.878036 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:14:40.878039 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:14:40.878459 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:14:40.878696 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0926 15:14:40.879046 (+ 350us) server_negotiation.cc:299] Negotiation successful
0926 15:14:40.879280 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260926 15:14:40.880831 2958 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.34.252" port: 45443 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 15:14:40 dist-test-slave-865v krb5kdc[2730](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435679, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.34.253@KRBTEST.COM
Sep 26 15:14:40 dist-test-slave-865v krb5kdc[2730](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435680, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.252@KRBTEST.COM for kudu/127.12.34.252@KRBTEST.COM
I20260926 15:14:40.894928 2965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:14:40.877723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41910 (local address 127.12.34.253:42593)
0926 15:14:40.878361 (+ 638us) server_negotiation.cc:206] Beginning negotiation
0926 15:14:40.878367 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:14:40.878392 (+ 25us) server_negotiation.cc:407] Connection header received
0926 15:14:40.878442 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:14:40.878445 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:14:40.878505 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:14:40.878644 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:14:40.880198 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:14:40.881367 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:14:40.883043 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:14:40.883442 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:14:40.883533 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:14:40.888302 (+ 4769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:14:40.888307 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:14:40.888310 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:14:40.888337 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:14:40.891932 (+ 3595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:14:40.892607 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:14:40.892616 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:14:40.892624 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:14:40.892781 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:14:40.893292 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:14:40.893298 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:14:40.893301 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:14:40.893709 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:14:40.893972 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0926 15:14:40.894250 (+ 278us) server_negotiation.cc:299] Negotiation successful
0926 15:14:40.894490 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":510,"thread_start_us":270,"threads_started":1}
I20260926 15:14:40.895700 2962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:14:40.881743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.252:45443 (local address 127.0.0.1:45454)
0926 15:14:40.881957 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0926 15:14:40.881972 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 15:14:40.882055 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:14:40.882370 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:14:40.882373 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:14:40.882710 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:14:40.882841 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:14:40.882849 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:14:40.884024 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:14:40.884027 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:14:40.885014 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:14:40.885025 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:14:40.886132 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:14:40.886136 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:14:40.886313 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:14:40.886881 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:14:40.886905 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:14:40.889010 (+ 2105us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:14:40.892687 (+ 3677us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:14:40.892690 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:14:40.892693 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:14:40.893194 (+ 501us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:14:40.893639 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:14:40.893642 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:14:40.893645 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:14:40.893795 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:14:40.894485 (+ 690us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:14:40.894489 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:14:40.894694 (+ 205us) client_negotiation.cc:769] Sending connection context
0926 15:14:40.894901 (+ 207us) client_negotiation.cc:240] Negotiation successful
0926 15:14:40.895226 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137}
I20260926 15:14:40.895859 2961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:14:40.881718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45454 (local address 127.12.34.252:45443)
0926 15:14:40.881917 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0926 15:14:40.881924 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:14:40.882041 (+ 117us) server_negotiation.cc:407] Connection header received
0926 15:14:40.882134 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:14:40.882140 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:14:40.882217 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:14:40.882324 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:14:40.882994 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:14:40.883875 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:14:40.885201 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:14:40.885953 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:14:40.886093 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:14:40.889241 (+ 3148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:14:40.889247 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:14:40.889253 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:14:40.889289 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:14:40.892497 (+ 3208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:14:40.893305 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:14:40.893310 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:14:40.893315 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:14:40.893486 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:14:40.893935 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:14:40.893941 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:14:40.893945 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:14:40.894333 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:14:40.894489 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0926 15:14:40.895022 (+ 533us) server_negotiation.cc:299] Negotiation successful
0926 15:14:40.895411 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":136,"server-negotiator.queue_time_us":74}
I20260926 15:14:40.900995 12427 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 15:14:40.902652 2958 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b2aee94e90c4de98e7319e5d9d7b3e7: Bootstrap starting.
I20260926 15:14:40.906086 2958 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7b2aee94e90c4de98e7319e5d9d7b3e7: Neither blocks nor log segments found. Creating new log.
I20260926 15:14:40.904069 2785 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "8b1323a33a7b40309605b4a0215dc594"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57296
I20260926 15:14:40.907176 2958 log.cc:824] T 00000000000000000000000000000000 P 7b2aee94e90c4de98e7319e5d9d7b3e7: Log is configured to *not* fsync() on all Append() calls
I20260926 15:14:40.910004 2958 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b2aee94e90c4de98e7319e5d9d7b3e7: No bootstrap required, opened a new log
I20260926 15:14:40.917955 2958 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7b2aee94e90c4de98e7319e5d9d7b3e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b1323a33a7b40309605b4a0215dc594" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 36943 } } peers { permanent_uuid: "edbd7d56c12245ef81d230093e189a24" member_type: VOTER last_known_addr { host: "127.12.34.253" port: 42593 } } peers { permanent_uuid: "7b2aee94e90c4de98e7319e5d9d7b3e7" member_type: VOTER last_known_addr { host: "127.12.34.252" port: 45443 } }
I20260926 15:14:40.918323 2958 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7b2aee94e90c4de98e7319e5d9d7b3e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b2aee94e90c4de98e7319e5d9d7b3e7, State: Initialized, Role: FOLLOWER
I20260926 15:14:40.918815 2958 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7b2aee94e90c4de98e7319e5d9d7b3e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b1323a33a7b40309605b4a0215dc594" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 36943 } } peers { permanent_uuid: "edbd7d56c12245ef81d230093e189a24" member_type: VOTER last_known_addr { host: "127.12.34.253" port: 42593 } } peers { permanent_uuid: "7b2aee94e90c4de98e7319e5d9d7b3e7" member_type: VOTER last_known_addr { host: "127.12.34.252" port: 45443 } }
I20260926 15:14:40.919965 2967 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b2aee94e90c4de98e7319e5d9d7b3e7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b1323a33a7b40309605b4a0215dc594" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 36943 } } peers { permanent_uuid: "edbd7d56c12245ef81d230093e189a24" member_type: VOTER last_known_addr { host: "127.12.34.253" port: 42593 } } peers { permanent_uuid: "7b2aee94e90c4de98e7319e5d9d7b3e7" member_type: VOTER last_known_addr { host: "127.12.34.252" port: 45443 } } }
I20260926 15:14:40.920454 2967 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b2aee94e90c4de98e7319e5d9d7b3e7 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 15:14:40.920495 2810 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.34.252" port: 45443 } attempt: 4
I20260926 15:14:40.921715 2958 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7b2aee94e90c4de98e7319e5d9d7b3e7 [sys.catalog]: configured and running, proceeding with master startup.
Sep 26 15:14:40 dist-test-slave-865v krb5kdc[2730](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435679, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.254@KRBTEST.COM for kudu/127.12.34.252@KRBTEST.COM
W20260926 15:14:40.932788 2978 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7b2aee94e90c4de98e7319e5d9d7b3e7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 15:14:40.933014 2978 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 15:14:40.939149 2812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:14:40.921309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.252:45443 (local address 127.0.0.1:45470)
0926 15:14:40.921597 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0926 15:14:40.921618 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 15:14:40.921786 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:14:40.922574 (+ 788us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:14:40.922579 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:14:40.923151 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:14:40.923410 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:14:40.923431 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:14:40.924786 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:14:40.924792 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:14:40.926536 (+ 1744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:14:40.926554 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:14:40.927658 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:14:40.927665 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:14:40.927830 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:14:40.928496 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:14:40.928546 (+ 50us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:14:40.931391 (+ 2845us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:14:40.936467 (+ 5076us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:14:40.936472 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:14:40.936476 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:14:40.936858 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:14:40.937314 (+ 456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:14:40.937319 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:14:40.937323 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:14:40.937467 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:14:40.938157 (+ 690us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:14:40.938161 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:14:40.938322 (+ 161us) client_negotiation.cc:769] Sending connection context
0926 15:14:40.938470 (+ 148us) client_negotiation.cc:240] Negotiation successful
0926 15:14:40.938760 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167}
I20260926 15:14:40.939172 2961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:14:40.922006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45470 (local address 127.12.34.252:45443)
0926 15:14:40.922230 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0926 15:14:40.922236 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:14:40.922255 (+ 19us) server_negotiation.cc:407] Connection header received
0926 15:14:40.922319 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:14:40.922323 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:14:40.922401 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:14:40.922508 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:14:40.923592 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:14:40.924621 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:14:40.926735 (+ 2114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:14:40.927494 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:14:40.927615 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:14:40.931579 (+ 3964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:14:40.931587 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:14:40.931594 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:14:40.931634 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:14:40.936336 (+ 4702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:14:40.936991 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:14:40.936997 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:14:40.937001 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:14:40.937192 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:14:40.937586 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:14:40.937592 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:14:40.937601 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:14:40.938032 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:14:40.938209 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0926 15:14:40.938565 (+ 356us) server_negotiation.cc:299] Negotiation successful
0926 15:14:40.938777 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 15:14:40.929826 2891 master_runner.cc:186] Error getting master registration for 127.12.34.254:36943: OK, instance_id { permanent_uuid: "8b1323a33a7b40309605b4a0215dc594" instance_seqno: 1790435679775489 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 15:14:40.942641 2891 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 15:14:40.944703 2810 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b1323a33a7b40309605b4a0215dc594: Bootstrap starting.
I20260926 15:14:40.948529 2810 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8b1323a33a7b40309605b4a0215dc594: Neither blocks nor log segments found. Creating new log.
I20260926 15:14:40.949599 2810 log.cc:824] T 00000000000000000000000000000000 P 8b1323a33a7b40309605b4a0215dc594: Log is configured to *not* fsync() on all Append() calls
I20260926 15:14:40.952195 2810 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b1323a33a7b40309605b4a0215dc594: No bootstrap required, opened a new log
I20260926 15:14:40.957413 2810 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8b1323a33a7b40309605b4a0215dc594 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b1323a33a7b40309605b4a0215dc594" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 36943 } } peers { permanent_uuid: "edbd7d56c12245ef81d230093e189a24" member_type: VOTER last_known_addr { host: "127.12.34.253" port: 42593 } } peers { permanent_uuid: "7b2aee94e90c4de98e7319e5d9d7b3e7" member_type: VOTER last_known_addr { host: "127.12.34.252" port: 45443 } }
I20260926 15:14:40.957791 2810 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8b1323a33a7b40309605b4a0215dc594 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b1323a33a7b40309605b4a0215dc594, State: Initialized, Role: FOLLOWER
I20260926 15:14:40.958272 2810 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8b1323a33a7b40309605b4a0215dc594 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b1323a33a7b40309605b4a0215dc594" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 36943 } } peers { permanent_uuid: "edbd7d56c12245ef81d230093e189a24" member_type: VOTER last_known_addr { host: "127.12.34.253" port: 42593 } } peers { permanent_uuid: "7b2aee94e90c4de98e7319e5d9d7b3e7" member_type: VOTER last_known_addr { host: "127.12.34.252" port: 45443 } }
I20260926 15:14:40.959143 2981 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b1323a33a7b40309605b4a0215dc594 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b1323a33a7b40309605b4a0215dc594" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 36943 } } peers { permanent_uuid: "edbd7d56c12245ef81d230093e189a24" member_type: VOTER last_known_addr { host: "127.12.34.253" port: 42593 } } peers { permanent_uuid: "7b2aee94e90c4de98e7319e5d9d7b3e7" member_type: VOTER last_known_addr { host: "127.12.34.252" port: 45443 } } }
I20260926 15:14:40.959507 2981 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b1323a33a7b40309605b4a0215dc594 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 15:14:40.960227 2810 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8b1323a33a7b40309605b4a0215dc594 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:14:40.960408 2785 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8b1323a33a7b40309605b4a0215dc594 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260926 15:14:40.960641 2785 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8b1323a33a7b40309605b4a0215dc594 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:14:40.964318 2785 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8b1323a33a7b40309605b4a0215dc594 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b1323a33a7b40309605b4a0215dc594" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 36943 } } peers { permanent_uuid: "edbd7d56c12245ef81d230093e189a24" member_type: VOTER last_known_addr { host: "127.12.34.253" port: 42593 } } peers { permanent_uuid: "7b2aee94e90c4de98e7319e5d9d7b3e7" member_type: VOTER last_known_addr { host: "127.12.34.252" port: 45443 } }
W20260926 15:14:40.970319 2992 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8b1323a33a7b40309605b4a0215dc594: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 15:14:40.970438 2992 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 15:14:40.969797 2743 master_runner.cc:186] Error getting master registration for 127.12.34.253:42593: OK, instance_id { permanent_uuid: "edbd7d56c12245ef81d230093e189a24" instance_seqno: 1790435680293273 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 15:14:40.981600 2743 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 15:14:40.981338 2785 leader_election.cc:288] T 00000000000000000000000000000000 P 8b1323a33a7b40309605b4a0215dc594 [CANDIDATE]: Term 1 election: Requested vote from peers edbd7d56c12245ef81d230093e189a24 (127.12.34.253:42593), 7b2aee94e90c4de98e7319e5d9d7b3e7 (127.12.34.252:45443)
I20260926 15:14:40.982010 2933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8b1323a33a7b40309605b4a0215dc594" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "7b2aee94e90c4de98e7319e5d9d7b3e7"
I20260926 15:14:40.982503 2933 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7b2aee94e90c4de98e7319e5d9d7b3e7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:14:40.986266 2933 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7b2aee94e90c4de98e7319e5d9d7b3e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b1323a33a7b40309605b4a0215dc594 in term 1.
I20260926 15:14:40.986935 2760 leader_election.cc:302] T 00000000000000000000000000000000 P 8b1323a33a7b40309605b4a0215dc594 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b2aee94e90c4de98e7319e5d9d7b3e7, 8b1323a33a7b40309605b4a0215dc594; no voters:
I20260926 15:14:40.987416 2981 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8b1323a33a7b40309605b4a0215dc594 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:14:40.993628 2981 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8b1323a33a7b40309605b4a0215dc594 [term 1 LEADER]: Becoming Leader. State: Replica: 8b1323a33a7b40309605b4a0215dc594, State: Running, Role: LEADER
I20260926 15:14:40.994251 2981 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8b1323a33a7b40309605b4a0215dc594 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b1323a33a7b40309605b4a0215dc594" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 36943 } } peers { permanent_uuid: "edbd7d56c12245ef81d230093e189a24" member_type: VOTER last_known_addr { host: "127.12.34.253" port: 42593 } } peers { permanent_uuid: "7b2aee94e90c4de98e7319e5d9d7b3e7" member_type: VOTER last_known_addr { host: "127.12.34.252" port: 45443 } }
I20260926 15:14:41.006831 2995 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b1323a33a7b40309605b4a0215dc594 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8b1323a33a7b40309605b4a0215dc594. Latest consensus state: current_term: 1 leader_uuid: "8b1323a33a7b40309605b4a0215dc594" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b1323a33a7b40309605b4a0215dc594" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 36943 } } peers { permanent_uuid: "edbd7d56c12245ef81d230093e189a24" member_type: VOTER last_known_addr { host: "127.12.34.253" port: 42593 } } peers { permanent_uuid: "7b2aee94e90c4de98e7319e5d9d7b3e7" member_type: VOTER last_known_addr { host: "127.12.34.252" port: 45443 } } }
I20260926 15:14:41.007436 2995 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b1323a33a7b40309605b4a0215dc594 [sys.catalog]: This master's current role is: LEADER
I20260926 15:14:41.014011 3000 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
W20260926 15:14:41.014473 2764 tablet.cc:2369] T 00000000000000000000000000000000 P 8b1323a33a7b40309605b4a0215dc594: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 15:14:41.018018 3000 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 15:14:41.028143 2884 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.34.252" port: 45443 } attempt: 5
I20260926 15:14:41.040710 2933 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7b2aee94e90c4de98e7319e5d9d7b3e7 [term 1 FOLLOWER]: Refusing update from remote peer 8b1323a33a7b40309605b4a0215dc594: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:14:41.041775 2995 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8b1323a33a7b40309605b4a0215dc594 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b2aee94e90c4de98e7319e5d9d7b3e7" member_type: VOTER last_known_addr { host: "127.12.34.252" port: 45443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 15:14:41.056496 2912 tablet.cc:2369] T 00000000000000000000000000000000 P 7b2aee94e90c4de98e7319e5d9d7b3e7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 15:14:41.061203 2967 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b2aee94e90c4de98e7319e5d9d7b3e7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8b1323a33a7b40309605b4a0215dc594. Latest consensus state: current_term: 1 leader_uuid: "8b1323a33a7b40309605b4a0215dc594" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b1323a33a7b40309605b4a0215dc594" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 36943 } } peers { permanent_uuid: "edbd7d56c12245ef81d230093e189a24" member_type: VOTER last_known_addr { host: "127.12.34.253" port: 42593 } } peers { permanent_uuid: "7b2aee94e90c4de98e7319e5d9d7b3e7" member_type: VOTER last_known_addr { host: "127.12.34.252" port: 45443 } } }
I20260926 15:14:41.061574 2967 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b2aee94e90c4de98e7319e5d9d7b3e7 [sys.catalog]: This master's current role is: FOLLOWER
Sep 26 15:14:41 dist-test-slave-865v krb5kdc[2730](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435680, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.34.253@KRBTEST.COM for kudu/127.12.34.252@KRBTEST.COM
I20260926 15:14:41.068655 2961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:14:41.021426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45472 (local address 127.12.34.252:45443)
0926 15:14:41.021607 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0926 15:14:41.021613 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:14:41.038284 (+ 16671us) server_negotiation.cc:407] Connection header received
0926 15:14:41.038655 (+ 371us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:14:41.038661 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:14:41.038745 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:14:41.038813 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:14:41.040936 (+ 2123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:14:41.042426 (+ 1490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:14:41.048300 (+ 5874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:14:41.048981 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:14:41.049072 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:14:41.057519 (+ 8447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:14:41.057526 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:14:41.057530 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:14:41.057567 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:14:41.061263 (+ 3696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:14:41.066129 (+ 4866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:14:41.066134 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:14:41.066137 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:14:41.066308 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:14:41.066801 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:14:41.066809 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:14:41.066815 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:14:41.067223 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:14:41.067475 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0926 15:14:41.068030 (+ 555us) server_negotiation.cc:299] Negotiation successful
0926 15:14:41.068272 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 15:14:41.076133 3004 mvcc.cc:205] Tried to move back new op lower bound from 7333624549509201920 to 7333624549359587328. Current Snapshot: MvccSnapshot[applied={T|T < 7333624549509201920}]
I20260926 15:14:41.077584 3005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:14:41.029520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45482 (local address 127.12.34.252:45443)
0926 15:14:41.051940 (+ 22420us) server_negotiation.cc:206] Beginning negotiation
0926 15:14:41.051946 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:14:41.051968 (+ 22us) server_negotiation.cc:407] Connection header received
0926 15:14:41.052014 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:14:41.052019 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:14:41.052086 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:14:41.053186 (+ 1100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:14:41.053460 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:14:41.054783 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:14:41.058785 (+ 4002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:14:41.059545 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:14:41.060256 (+ 711us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:14:41.065052 (+ 4796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:14:41.065060 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:14:41.065065 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:14:41.065133 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:14:41.073629 (+ 8496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:14:41.075265 (+ 1636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:14:41.075271 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:14:41.075277 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:14:41.075436 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:14:41.075892 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:14:41.075896 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:14:41.075899 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:14:41.076221 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:14:41.076778 (+ 557us) server_negotiation.cc:1035] Waiting for connection context
0926 15:14:41.076908 (+ 130us) server_negotiation.cc:299] Negotiation successful
0926 15:14:41.077134 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22277,"thread_start_us":22629,"threads_started":1}
I20260926 15:14:41.077797 2887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:14:41.028752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.34.252:45443 (local address 127.0.0.1:45482)
0926 15:14:41.029030 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0926 15:14:41.029053 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 15:14:41.029199 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:14:41.052324 (+ 23125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:14:41.052329 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:14:41.052828 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:14:41.053027 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:14:41.053043 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:14:41.057051 (+ 4008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:14:41.057056 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:14:41.058551 (+ 1495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:14:41.058565 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:14:41.059738 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:14:41.059745 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:14:41.059983 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:14:41.061896 (+ 1913us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:14:41.061922 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:14:41.064828 (+ 2906us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:14:41.073796 (+ 8968us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:14:41.073808 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:14:41.073813 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:14:41.074235 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:14:41.075571 (+ 1336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:14:41.075578 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:14:41.075584 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:14:41.075753 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:14:41.076349 (+ 596us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:14:41.076355 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:14:41.076509 (+ 154us) client_negotiation.cc:769] Sending connection context
0926 15:14:41.076653 (+ 144us) client_negotiation.cc:240] Negotiation successful
0926 15:14:41.077387 (+ 734us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"mutex_wait_us":384}
I20260926 15:14:41.079617 2965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:14:41.021448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41926 (local address 127.12.34.253:42593)
0926 15:14:41.021646 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0926 15:14:41.021651 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:14:41.045486 (+ 23835us) server_negotiation.cc:407] Connection header received
0926 15:14:41.045956 (+ 470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:14:41.045964 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:14:41.046066 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:14:41.046151 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:14:41.050126 (+ 3975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:14:41.051119 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:14:41.055674 (+ 4555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:14:41.056338 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:14:41.056435 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:14:41.071491 (+ 15056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:14:41.071499 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:14:41.071506 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:14:41.071548 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:14:41.075047 (+ 3499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:14:41.075859 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:14:41.075865 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:14:41.075871 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:14:41.075992 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:14:41.076488 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:14:41.076494 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:14:41.076496 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:14:41.076829 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:14:41.077047 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0926 15:14:41.079091 (+ 2044us) server_negotiation.cc:299] Negotiation successful
0926 15:14:41.079307 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":58}
I20260926 15:14:41.083698 2884 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P edbd7d56c12245ef81d230093e189a24: Bootstrap starting.
I20260926 15:14:41.087141 2884 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P edbd7d56c12245ef81d230093e189a24: Neither blocks nor log segments found. Creating new log.
I20260926 15:14:41.088339 2884 log.cc:824] T 00000000000000000000000000000000 P edbd7d56c12245ef81d230093e189a24: Log is configured to *not* fsync() on all Append() calls
I20260926 15:14:41.093966 2964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:14:41.019014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57304 (local address 127.12.34.254:36943)
0926 15:14:41.019204 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0926 15:14:41.019212 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 15:14:41.035156 (+ 15944us) server_negotiation.cc:407] Connection header received
0926 15:14:41.035626 (+ 470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:14:41.035632 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:14:41.035734 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:14:41.035809 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:14:41.050511 (+ 14702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:14:41.051614 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:14:41.055009 (+ 3395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:14:41.055690 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:14:41.055787 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:14:41.065397 (+ 9610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:14:41.065404 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:14:41.065408 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:14:41.065448 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:14:41.069181 (+ 3733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:14:41.090051 (+ 20870us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:14:41.090058 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:14:41.090065 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:14:41.090241 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:14:41.091415 (+ 1174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:14:41.091420 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:14:41.091423 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:14:41.091801 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:14:41.092051 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0926 15:14:41.093350 (+ 1299us) server_negotiation.cc:299] Negotiation successful
0926 15:14:41.093612 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 15:14:41.104875 2981 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b1323a33a7b40309605b4a0215dc594 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8b1323a33a7b40309605b4a0215dc594" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b1323a33a7b40309605b4a0215dc594" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 36943 } } peers { permanent_uuid: "edbd7d56c12245ef81d230093e189a24" member_type: VOTER last_known_addr { host: "127.12.34.253" port: 42593 } } peers { permanent_uuid: "7b2aee94e90c4de98e7319e5d9d7b3e7" member_type: VOTER last_known_addr { host: "127.12.34.252" port: 45443 } } }
I20260926 15:14:41.105225 2981 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b1323a33a7b40309605b4a0215dc594 [sys.catalog]: This master's current role is: LEADER
I20260926 15:14:41.105964 3000 catalog_manager.cc:1347] Generated new cluster ID: 8501628b59d84c2897e04bd7ed02590e
I20260926 15:14:41.106065 3000 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 15:14:41.122721 2967 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b2aee94e90c4de98e7319e5d9d7b3e7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8b1323a33a7b40309605b4a0215dc594" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b1323a33a7b40309605b4a0215dc594" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 36943 } } peers { permanent_uuid: "edbd7d56c12245ef81d230093e189a24" member_type: VOTER last_known_addr { host: "127.12.34.253" port: 42593 } } peers { permanent_uuid: "7b2aee94e90c4de98e7319e5d9d7b3e7" member_type: VOTER last_known_addr { host: "127.12.34.252" port: 45443 } } }
I20260926 15:14:41.122982 2967 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b2aee94e90c4de98e7319e5d9d7b3e7 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 15:14:41.134619 2884 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P edbd7d56c12245ef81d230093e189a24: No bootstrap required, opened a new log
I20260926 15:14:41.149291 3000 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 15:14:41.152848 3000 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:14:41.155360 2884 raft_consensus.cc:348] T 00000000000000000000000000000000 P edbd7d56c12245ef81d230093e189a24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b1323a33a7b40309605b4a0215dc594" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 36943 } } peers { permanent_uuid: "edbd7d56c12245ef81d230093e189a24" member_type: VOTER last_known_addr { host: "127.12.34.253" port: 42593 } } peers { permanent_uuid: "7b2aee94e90c4de98e7319e5d9d7b3e7" member_type: VOTER last_known_addr { host: "127.12.34.252" port: 45443 } }
I20260926 15:14:41.155759 2884 raft_consensus.cc:729] T 00000000000000000000000000000000 P edbd7d56c12245ef81d230093e189a24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edbd7d56c12245ef81d230093e189a24, State: Initialized, Role: FOLLOWER
I20260926 15:14:41.156255 2884 consensus_queue.cc:260] T 00000000000000000000000000000000 P edbd7d56c12245ef81d230093e189a24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b1323a33a7b40309605b4a0215dc594" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 36943 } } peers { permanent_uuid: "edbd7d56c12245ef81d230093e189a24" member_type: VOTER last_known_addr { host: "127.12.34.253" port: 42593 } } peers { permanent_uuid: "7b2aee94e90c4de98e7319e5d9d7b3e7" member_type: VOTER last_known_addr { host: "127.12.34.252" port: 45443 } }
I20260926 15:14:41.173614 3011 sys_catalog.cc:455] T 00000000000000000000000000000000 P edbd7d56c12245ef81d230093e189a24 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b1323a33a7b40309605b4a0215dc594" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 36943 } } peers { permanent_uuid: "edbd7d56c12245ef81d230093e189a24" member_type: VOTER last_known_addr { host: "127.12.34.253" port: 42593 } } peers { permanent_uuid: "7b2aee94e90c4de98e7319e5d9d7b3e7" member_type: VOTER last_known_addr { host: "127.12.34.252" port: 45443 } } }
I20260926 15:14:41.174063 3011 sys_catalog.cc:458] T 00000000000000000000000000000000 P edbd7d56c12245ef81d230093e189a24 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 15:14:41.178889 2884 sys_catalog.cc:564] T 00000000000000000000000000000000 P edbd7d56c12245ef81d230093e189a24 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:14:41.179311 2858 raft_consensus.cc:3037] T 00000000000000000000000000000000 P edbd7d56c12245ef81d230093e189a24 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:14:41.180918 2859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8b1323a33a7b40309605b4a0215dc594" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "edbd7d56c12245ef81d230093e189a24"
I20260926 15:14:41.187443 2858 raft_consensus.cc:1261] T 00000000000000000000000000000000 P edbd7d56c12245ef81d230093e189a24 [term 1 FOLLOWER]: Refusing update from remote peer 8b1323a33a7b40309605b4a0215dc594: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:14:41.190958 2859 raft_consensus.cc:2454] T 00000000000000000000000000000000 P edbd7d56c12245ef81d230093e189a24 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b1323a33a7b40309605b4a0215dc594 in term 1.
I20260926 15:14:41.193174 2995 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8b1323a33a7b40309605b4a0215dc594 [LEADER]: Connected to new peer: Peer: permanent_uuid: "edbd7d56c12245ef81d230093e189a24" member_type: VOTER last_known_addr { host: "127.12.34.253" port: 42593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:14:41.200299 3000 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8b1323a33a7b40309605b4a0215dc594: Generated new TSK 0
I20260926 15:14:41.201582 3000 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 15:14:41.230212 3011 sys_catalog.cc:455] T 00000000000000000000000000000000 P edbd7d56c12245ef81d230093e189a24 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8b1323a33a7b40309605b4a0215dc594. Latest consensus state: current_term: 1 leader_uuid: "8b1323a33a7b40309605b4a0215dc594" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b1323a33a7b40309605b4a0215dc594" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 36943 } } peers { permanent_uuid: "edbd7d56c12245ef81d230093e189a24" member_type: VOTER last_known_addr { host: "127.12.34.253" port: 42593 } } peers { permanent_uuid: "7b2aee94e90c4de98e7319e5d9d7b3e7" member_type: VOTER last_known_addr { host: "127.12.34.252" port: 45443 } } }
I20260926 15:14:41.230618 3011 sys_catalog.cc:458] T 00000000000000000000000000000000 P edbd7d56c12245ef81d230093e189a24 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 15:14:41.230793 3018 sys_catalog.cc:455] T 00000000000000000000000000000000 P edbd7d56c12245ef81d230093e189a24 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8b1323a33a7b40309605b4a0215dc594" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b1323a33a7b40309605b4a0215dc594" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 36943 } } peers { permanent_uuid: "edbd7d56c12245ef81d230093e189a24" member_type: VOTER last_known_addr { host: "127.12.34.253" port: 42593 } } peers { permanent_uuid: "7b2aee94e90c4de98e7319e5d9d7b3e7" member_type: VOTER last_known_addr { host: "127.12.34.252" port: 45443 } } }
I20260926 15:14:41.231166 3018 sys_catalog.cc:458] T 00000000000000000000000000000000 P edbd7d56c12245ef81d230093e189a24 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 15:14:41.259686 3028 catalog_manager.cc:1259] Loaded cluster ID: 8501628b59d84c2897e04bd7ed02590e
I20260926 15:14:41.259804 3028 catalog_manager.cc:1552] T 00000000000000000000000000000000 P edbd7d56c12245ef81d230093e189a24: loading cluster ID for follower catalog manager: success
I20260926 15:14:41.263674 3028 catalog_manager.cc:1574] T 00000000000000000000000000000000 P edbd7d56c12245ef81d230093e189a24: acquiring CA information for follower catalog manager: success
I20260926 15:14:41.266206 3028 catalog_manager.cc:1602] T 00000000000000000000000000000000 P edbd7d56c12245ef81d230093e189a24: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260926 15:14:41.398172 2964 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:14:41.371085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57314 (local address 127.12.34.254:36943)
0926 15:14:41.371370 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0926 15:14:41.371380 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 15:14:41.396036 (+ 24656us) server_negotiation.cc:407] Connection header received
0926 15:14:41.396266 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:14:41.396275 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:14:41.396385 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:14:41.396501 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:14:41.397887 (+ 1386us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57314: BlockingRecv error: recv got EOF from 127.0.0.1:57314 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
W20260926 15:14:41.400378 2965 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:14:41.378304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41930 (local address 127.12.34.253:42593)
0926 15:14:41.378536 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0926 15:14:41.378545 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 15:14:41.398763 (+ 20218us) server_negotiation.cc:407] Connection header received
0926 15:14:41.398933 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:14:41.398938 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:14:41.399019 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:14:41.399119 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:14:41.400058 (+ 939us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41930: BlockingRecv error: recv got EOF from 127.0.0.1:41930 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
W20260926 15:14:41.401897 3005 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:14:41.378727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45496 (local address 127.12.34.252:45443)
0926 15:14:41.378941 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0926 15:14:41.378949 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 15:14:41.400153 (+ 21204us) server_negotiation.cc:407] Connection header received
0926 15:14:41.400419 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:14:41.400426 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:14:41.400548 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:14:41.400645 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:14:41.401678 (+ 1033us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45496: BlockingRecv error: recv got EOF from 127.0.0.1:45496 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 15:14:41.457132 2965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:14:41.442065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41938 (local address 127.12.34.253:42593)
0926 15:14:41.442268 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0926 15:14:41.442275 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:14:41.449100 (+ 6825us) server_negotiation.cc:407] Connection header received
0926 15:14:41.449615 (+ 515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:14:41.449622 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:14:41.449847 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:14:41.449983 (+ 136us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 15:14:41.450684 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:14:41.451847 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:14:41.453909 (+ 2062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:14:41.454781 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:14:41.454889 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:14:41.455627 (+ 738us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 15:14:41.456258 (+ 631us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 15:14:41.456368 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0926 15:14:41.456609 (+ 241us) server_negotiation.cc:299] Negotiation successful
0926 15:14:41.456704 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260926 15:14:41.504292 3005 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:14:41.490409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45512 (local address 127.12.34.252:45443)
0926 15:14:41.490589 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0926 15:14:41.490595 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:14:41.501636 (+ 11041us) server_negotiation.cc:407] Connection header received
0926 15:14:41.502550 (+ 914us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:14:41.502558 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:14:41.502716 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:14:41.502805 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:14:41.504002 (+ 1197us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45512: BlockingRecv error: recv got EOF from 127.0.0.1:45512 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 15:14:41.549131 2965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:14:41.535134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41942 (local address 127.12.34.253:42593)
0926 15:14:41.535351 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0926 15:14:41.535357 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:14:41.542187 (+ 6830us) server_negotiation.cc:407] Connection header received
0926 15:14:41.542684 (+ 497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:14:41.542691 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:14:41.542801 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:14:41.542870 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 15:14:41.543532 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:14:41.544654 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:14:41.546745 (+ 2091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:14:41.547470 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:14:41.547596 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:14:41.548262 (+ 666us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 15:14:41.548409 (+ 147us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 15:14:41.548469 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 15:14:41.548687 (+ 218us) server_negotiation.cc:299] Negotiation successful
0926 15:14:41.548778 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20260926 15:14:41.598356 3005 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:14:41.583740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45518 (local address 127.12.34.252:45443)
0926 15:14:41.583941 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0926 15:14:41.583948 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:14:41.596214 (+ 12266us) server_negotiation.cc:407] Connection header received
0926 15:14:41.596741 (+ 527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:14:41.596749 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:14:41.596857 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:14:41.596944 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:14:41.598101 (+ 1157us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45518: BlockingRecv error: recv got EOF from 127.0.0.1:45518 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 15:14:41.640870 2965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:14:41.632608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41952 (local address 127.12.34.253:42593)
0926 15:14:41.632802 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0926 15:14:41.632808 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:14:41.634981 (+ 2173us) server_negotiation.cc:407] Connection header received
0926 15:14:41.635181 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:14:41.635187 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:14:41.635278 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:14:41.635386 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 15:14:41.635865 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:14:41.637026 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:14:41.638879 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:14:41.639533 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:14:41.639653 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:14:41.640099 (+ 446us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 15:14:41.640235 (+ 136us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 15:14:41.640326 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 15:14:41.640489 (+ 163us) server_negotiation.cc:299] Negotiation successful
0926 15:14:41.640569 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260926 15:14:41.652426 3005 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:14:41.650616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45524 (local address 127.12.34.252:45443)
0926 15:14:41.650825 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0926 15:14:41.650831 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:14:41.651058 (+ 227us) server_negotiation.cc:407] Connection header received
0926 15:14:41.651205 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:14:41.651210 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:14:41.651295 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:14:41.651404 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:14:41.652182 (+ 778us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45524: BlockingRecv error: recv got EOF from 127.0.0.1:45524 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 15:14:41.667148 2965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:14:41.660695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41966 (local address 127.12.34.253:42593)
0926 15:14:41.660986 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0926 15:14:41.660992 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:14:41.661512 (+ 520us) server_negotiation.cc:407] Connection header received
0926 15:14:41.661713 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:14:41.661718 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:14:41.661809 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:14:41.661888 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 15:14:41.662419 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:14:41.663208 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:14:41.665394 (+ 2186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:14:41.665812 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:14:41.665905 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:14:41.666402 (+ 497us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 15:14:41.666548 (+ 146us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 15:14:41.666618 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 15:14:41.666800 (+ 182us) server_negotiation.cc:299] Negotiation successful
0926 15:14:41.666865 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120}
W20260926 15:14:41.674700 3005 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:14:41.672922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45538 (local address 127.12.34.252:45443)
0926 15:14:41.673180 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0926 15:14:41.673187 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:14:41.673470 (+ 283us) server_negotiation.cc:407] Connection header received
0926 15:14:41.673615 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:14:41.673620 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:14:41.673709 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:14:41.673816 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:14:41.674468 (+ 652us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45538: BlockingRecv error: recv got EOF from 127.0.0.1:45538 (error 108)
Metrics: {"server-negotiator.queue_time_us":106}
I20260926 15:14:41.681797 2995 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b1323a33a7b40309605b4a0215dc594 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8b1323a33a7b40309605b4a0215dc594" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b1323a33a7b40309605b4a0215dc594" member_type: VOTER last_known_addr { host: "127.12.34.254" port: 36943 } } peers { permanent_uuid: "edbd7d56c12245ef81d230093e189a24" member_type: VOTER last_known_addr { host: "127.12.34.253" port: 42593 } } peers { permanent_uuid: "7b2aee94e90c4de98e7319e5d9d7b3e7" member_type: VOTER last_known_addr { host: "127.12.34.252" port: 45443 } } }
I20260926 15:14:41.682111 2995 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b1323a33a7b40309605b4a0215dc594 [sys.catalog]: This master's current role is: LEADER
I20260926 15:14:41.693346 2965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:14:41.686722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41974 (local address 127.12.34.253:42593)
0926 15:14:41.686952 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0926 15:14:41.686959 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:14:41.687290 (+ 331us) server_negotiation.cc:407] Connection header received
0926 15:14:41.687496 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:14:41.687500 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:14:41.687572 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:14:41.687661 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 15:14:41.688180 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:14:41.688959 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:14:41.691261 (+ 2302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:14:41.691749 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:14:41.691862 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:14:41.692366 (+ 504us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 15:14:41.692469 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 15:14:41.692537 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 15:14:41.692787 (+ 250us) server_negotiation.cc:299] Negotiation successful
0926 15:14:41.692895 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
W20260926 15:14:41.701398 3005 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:14:41.699318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45550 (local address 127.12.34.252:45443)
0926 15:14:41.699489 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0926 15:14:41.699493 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:14:41.699955 (+ 462us) server_negotiation.cc:407] Connection header received
0926 15:14:41.700119 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:14:41.700133 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:14:41.700234 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:14:41.700349 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:14:41.701117 (+ 768us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45550: BlockingRecv error: recv got EOF from 127.0.0.1:45550 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 15:14:41.727766 2965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:14:41.721271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41980 (local address 127.12.34.253:42593)
0926 15:14:41.721471 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0926 15:14:41.721476 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:14:41.721744 (+ 268us) server_negotiation.cc:407] Connection header received
0926 15:14:41.721975 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:14:41.721978 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:14:41.722045 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:14:41.722122 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 15:14:41.722721 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:14:41.723612 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:14:41.725832 (+ 2220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:14:41.726265 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:14:41.726364 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:14:41.726940 (+ 576us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 15:14:41.727053 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 15:14:41.727153 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 15:14:41.727364 (+ 211us) server_negotiation.cc:299] Negotiation successful
0926 15:14:41.727428 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
W20260926 15:14:41.736173 3005 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:14:41.733864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45560 (local address 127.12.34.252:45443)
0926 15:14:41.734085 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0926 15:14:41.734091 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:14:41.734437 (+ 346us) server_negotiation.cc:407] Connection header received
0926 15:14:41.734642 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:14:41.734650 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:14:41.734810 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:14:41.734901 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:14:41.735876 (+ 975us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45560: BlockingRecv error: recv got EOF from 127.0.0.1:45560 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
I20260926 15:14:41.778267 2965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:14:41.772103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41986 (local address 127.12.34.253:42593)
0926 15:14:41.772344 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0926 15:14:41.772351 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:14:41.772713 (+ 362us) server_negotiation.cc:407] Connection header received
0926 15:14:41.772872 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:14:41.772878 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:14:41.772967 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:14:41.773116 (+ 149us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 15:14:41.773531 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:14:41.774682 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:14:41.776104 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:14:41.776754 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:14:41.776892 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:14:41.777376 (+ 484us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 15:14:41.777539 (+ 163us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 15:14:41.777649 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0926 15:14:41.777838 (+ 189us) server_negotiation.cc:299] Negotiation successful
0926 15:14:41.777918 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
W20260926 15:14:41.791607 3005 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:14:41.785516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45574 (local address 127.12.34.252:45443)
0926 15:14:41.785701 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0926 15:14:41.785707 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:14:41.789969 (+ 4262us) server_negotiation.cc:407] Connection header received
0926 15:14:41.790253 (+ 284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:14:41.790259 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:14:41.790357 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:14:41.790458 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:14:41.791382 (+ 924us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45574: BlockingRecv error: recv got EOF from 127.0.0.1:45574 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 15:14:41.865934 2965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:14:41.859289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41996 (local address 127.12.34.253:42593)
0926 15:14:41.859520 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0926 15:14:41.859527 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:14:41.860098 (+ 571us) server_negotiation.cc:407] Connection header received
0926 15:14:41.860292 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:14:41.860297 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:14:41.860375 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:14:41.860554 (+ 179us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 15:14:41.861041 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:14:41.862200 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:14:41.864076 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:14:41.864561 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:14:41.864658 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:14:41.865123 (+ 465us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 15:14:41.865265 (+ 142us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 15:14:41.865335 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 15:14:41.865484 (+ 149us) server_negotiation.cc:299] Negotiation successful
0926 15:14:41.865548 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
W20260926 15:14:41.874083 3005 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:14:41.872089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45576 (local address 127.12.34.252:45443)
0926 15:14:41.872330 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0926 15:14:41.872337 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:14:41.872583 (+ 246us) server_negotiation.cc:407] Connection header received
0926 15:14:41.872756 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:14:41.872761 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:14:41.872836 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:14:41.872934 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:14:41.873834 (+ 900us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45576: BlockingRecv error: recv got EOF from 127.0.0.1:45576 (error 108)
Metrics: {"server-negotiator.queue_time_us":96}
I20260926 15:14:41.935041 2978 catalog_manager.cc:1259] Loaded cluster ID: 8501628b59d84c2897e04bd7ed02590e
I20260926 15:14:41.935143 2978 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7b2aee94e90c4de98e7319e5d9d7b3e7: loading cluster ID for follower catalog manager: success
I20260926 15:14:41.937625 2978 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7b2aee94e90c4de98e7319e5d9d7b3e7: acquiring CA information for follower catalog manager: success
I20260926 15:14:41.939347 2978 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7b2aee94e90c4de98e7319e5d9d7b3e7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 15:14:42.011719 2965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:14:42.006737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42000 (local address 127.12.34.253:42593)
0926 15:14:42.006927 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0926 15:14:42.006931 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:14:42.007291 (+ 360us) server_negotiation.cc:407] Connection header received
0926 15:14:42.007425 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:14:42.007429 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:14:42.007501 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:14:42.007594 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 15:14:42.007955 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:14:42.008682 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:14:42.010066 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:14:42.010489 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:14:42.010616 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:14:42.011046 (+ 430us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 15:14:42.011146 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 15:14:42.011229 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 15:14:42.011432 (+ 203us) server_negotiation.cc:299] Negotiation successful
0926 15:14:42.011498 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 15:14:42.024082 3005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:14:42.017506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45580 (local address 127.12.34.252:45443)
0926 15:14:42.017716 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0926 15:14:42.017723 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:14:42.018119 (+ 396us) server_negotiation.cc:407] Connection header received
0926 15:14:42.018354 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:14:42.018358 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:14:42.018511 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:14:42.018645 (+ 134us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 15:14:42.019107 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:14:42.019875 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:14:42.021421 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:14:42.021906 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:14:42.022048 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:14:42.022653 (+ 605us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 15:14:42.023326 (+ 673us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 15:14:42.023510 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0926 15:14:42.023569 (+ 59us) server_negotiation.cc:299] Negotiation successful
0926 15:14:42.023656 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 15:14:42.039844 2965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:14:42.030205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42012 (local address 127.12.34.253:42593)
0926 15:14:42.030418 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0926 15:14:42.030428 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 15:14:42.033825 (+ 3397us) server_negotiation.cc:407] Connection header received
0926 15:14:42.034095 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:14:42.034101 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:14:42.034202 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:14:42.034308 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 15:14:42.035062 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:14:42.036186 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:14:42.037929 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:14:42.038526 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:14:42.038660 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:14:42.039044 (+ 384us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 15:14:42.039169 (+ 125us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 15:14:42.039262 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 15:14:42.039434 (+ 172us) server_negotiation.cc:299] Negotiation successful
0926 15:14:42.039512 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 15:14:42.040051 3134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:14:42.030078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57318 (local address 127.12.34.254:36943)
0926 15:14:42.032392 (+ 2314us) server_negotiation.cc:206] Beginning negotiation
0926 15:14:42.032401 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 15:14:42.032914 (+ 513us) server_negotiation.cc:407] Connection header received
0926 15:14:42.033038 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:14:42.033045 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:14:42.033415 (+ 370us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:14:42.033943 (+ 528us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 15:14:42.034382 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:14:42.035381 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:14:42.037152 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:14:42.037831 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:14:42.037968 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:14:42.038511 (+ 543us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 15:14:42.039015 (+ 504us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 15:14:42.039121 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 15:14:42.039640 (+ 519us) server_negotiation.cc:299] Negotiation successful
0926 15:14:42.039734 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2143,"thread_start_us":2001,"threads_started":1}
I20260926 15:14:42.040334 3005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:14:42.030535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45588 (local address 127.12.34.252:45443)
0926 15:14:42.030727 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0926 15:14:42.030732 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:14:42.033177 (+ 2445us) server_negotiation.cc:407] Connection header received
0926 15:14:42.033288 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:14:42.033296 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:14:42.033409 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:14:42.033509 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 15:14:42.034829 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:14:42.035994 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:14:42.038247 (+ 2253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:14:42.038900 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:14:42.039025 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:14:42.039520 (+ 495us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 15:14:42.039642 (+ 122us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 15:14:42.039704 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 15:14:42.039934 (+ 230us) server_negotiation.cc:299] Negotiation successful
0926 15:14:42.040010 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 15:14:42.045367 12427 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 2743
I20260926 15:14:42.046320 2805 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:14:42.159654 12427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8uoJGv/build/asan/bin/kudu with pid 2743
W20260926 15:14:42.180495 2835 connection.cc:535] server connection from 127.0.0.1:41892 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 15:14:42.180882 12427 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 2817
I20260926 15:14:42.181962 2879 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:14:42.289170 12427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8uoJGv/build/asan/bin/kudu with pid 2817
I20260926 15:14:42.309064 12427 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8uoJGv/build/asan/bin/kudu pid 2891
I20260926 15:14:42.310192 2953 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 15:14:42.424119 12427 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8uoJGv/build/asan/bin/kudu with pid 2891
2026-09-26T15:14:42Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4441 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (9511 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (364112 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST